Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives

• URL: http://arxiv.org/abs/2410.05558v2
• Date: Sun, 17 Nov 2024 17:00:11 GMT
• Title: Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives
• Authors: Xinliang Frederick Zhang, Nick Beauchamp, Lu Wang,
• Abstract summary: We study an essential task of temporal reasoning -- temporal graph generation. We show that this task presents great challenges even for the most powerful language models. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought.
• Score: 6.631626634132574
• License:
• Abstract: Reasoning about time and temporal relations is an integral aspect of human cognition, essential for perceiving the world and navigating our experiences. Though large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, temporal reasoning remains challenging due to its intrinsic complexity. In this work, we first study an essential task of temporal reasoning -- temporal graph generation, to unveil LLMs' inherent, global reasoning capabilities. We show that this task presents great challenges even for the most powerful LLMs, such as GPT-3.5/4. We also notice a significant performance gap by small models (<10B) that lag behind LLMs by 50%. Next, we study how to close this gap with a budget constraint, e.g., not using model finetuning. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph. Extensive experiments showcase the efficacy of NoT in improving various metrics. Notably, NoT attains the highest F1 on the Schema-11 evaluation set, while securing an overall F1 on par with GPT-3.5. NoT also achieves the best structural similarity across the board, even compared with GPT-3.5/4. Our code is available at https://github.com/launchnlp/NoT.

Related papers

• The Jumping Reasoning Curve? Tracking the Evolution of Reasoning Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles [29.214813685163218]
  OpenAI's releases of o1 and o3 mark a paradigm shift in Large Language Models towards advanced reasoning capabilities. We track the evolution of the GPT-[n] and o-[n] series models on challenging multimodal puzzles. The superior performance of o1 comes at nearly 750 times the computational cost of GPT-4o, raising concerns about its efficiency.
  arXiv  Detail & Related papers  (2025-02-03T05:47:04Z)
• Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling [52.34735382627312]
  Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. Existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. We present T1 to scale reinforcement learning by encouraging exploration and understand inference scaling.
  arXiv  Detail & Related papers  (2025-01-20T18:33:33Z)
• ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events [0.20132569095596248]
  We present ChronoSense, a new benchmark for evaluating Large Language Models' temporal understanding. We assess the performance of seven recent LLMs using this benchmark and the results indicate that models handle Allen relations, even symmetrical ones, quite differently. Overall, the models' low performance highlights the need for improved temporal understanding in LLMs.
  arXiv  Detail & Related papers  (2025-01-06T14:27:41Z)
• Failure Modes of LLMs for Causal Reasoning on Narratives [51.19592551510628]
  We investigate the causal reasoning abilities of large language models (LLMs) through the representative problem of inferring causal relationships from narratives. We find that even state-of-the-art language models rely on unreliable shortcuts, both in terms of the narrative presentation and their parametric knowledge.
  arXiv  Detail & Related papers  (2024-10-31T12:48:58Z)
• Timo: Towards Better Temporal Reasoning for Language Models [38.27548375148604]
  Reasoning about time is essential for Large Language Models to understand the world. We build a universal framework to handle a variety of temporal reasoning tasks. We develop Timo, a model designed to excel in temporal reasoning at the 7B and 13B scales.
  arXiv  Detail & Related papers  (2024-06-20T10:52:14Z)
• Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning? [70.19200858203388]
  Temporal reasoning is fundamental for large language models to comprehend the world. CoTempQA is a benchmark containing four co-temporal scenarios. Our experiments reveal a significant gap between the performance of current LLMs and human-level reasoning.
  arXiv  Detail & Related papers  (2024-06-13T12:56:21Z)
• Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency [53.8779374188643]
  We propose a principled framework with provable regret guarantees to orchestrate reasoning and acting. Specifically, we design a prompt template for reasoning that learns from the memory buffer and plans a future trajectory over a long horizon. At each step, the LLM agent takes the initial action of the planned trajectory ("act for now"), stores the collected feedback in the memory buffer, and reinvokes the reasoning routine to replan the future trajectory from the new state.
  arXiv  Detail & Related papers  (2023-09-29T16:36:39Z)
• Generic Temporal Reasoning with Differential Analysis and Explanation [61.96034987217583]
  We introduce a novel task named TODAY that bridges the gap with temporal differential analysis. TODAY evaluates whether systems can correctly understand the effect of incremental changes. We show that TODAY's supervision style and explanation annotations can be used in joint learning.
  arXiv  Detail & Related papers  (2022-12-20T17:40:03Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)
• TIMEDIAL: Temporal Commonsense Reasoning in Dialog [43.24596551545824]
  We present the first study to investigate pre-trained language models for their temporal reasoning capabilities in dialogs. We formulate TIME-DIAL as a multiple-choice cloze task with over 1.1K carefully curated dialogs. Empirical results demonstrate that even the best performing models struggle on this task compared to humans.
  arXiv  Detail & Related papers  (2021-06-08T17:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.