Speculate with Memory: Lossless Acceleration for LLM Agents

• URL: http://arxiv.org/abs/2607.12236v1
• Date: Tue, 14 Jul 2026 00:36:31 GMT
• Title: Speculate with Memory: Lossless Acceleration for LLM Agents
• Abstract summary: Speculative execution accelerates LLM agents by using a smaller, cheaper model to predict and pre-launch the next step while the environment is idle.<n>We equip the speculator with three online memory systems that learn from past agent trajectories.<n>We evaluate this approach on six benchmarks spanning three speculation types: action prediction, observation prediction, and chained prediction.
• Score: 41.48103818220709
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speculative execution accelerates LLM agents by using a smaller, cheaper model to predict and pre-launch the next step while the environment is idle. However, existing speculators are stateless and discard all information between tasks, preventing prediction quality from improving with experience. We equip the speculator with three online memory systems that learn from past agent trajectories: a contrastive transition table tracking action-sequence statistics, an episodic memory retrieving contextually similar segments, and a confusion tracker suppressing recurring errors. We evaluate this approach on six benchmarks spanning three speculation types: action prediction, observation prediction, and chained prediction. Memory-augmented speculation yields a 19--39\% relative accuracy improvement on action prediction and up to a $2.5\times$ increase on observation prediction tasks with repetitive action spaces. These gains grow continuously as memory accumulates and generalize across speculator models of varying cost. All speculation is lossless because it runs during idle time at zero added wall-clock cost, and the actor's trajectory is identical to non-speculative execution.

Related papers

• Repurposing Deep Limit Order Book Forecasting for Scenario-Conditioned Market Impact Modeling [49.590388401787834]
  We introduce a model-agnostic framework that compares a trained forecaster's predictive distributions before and after injecting mechanically valid counterfactual messages.<n>A Transformer-based forecaster recovered scenario rankings with a Spearman correlation of 0.99 and 97.2% directional agreement with realized historical outcomes.
  arXiv  Detail & Related papers  (2026-09-15T10:06:03Z)
• Uncertainty-Guided Sparse Refinement for Action Chunking Transformer Policies [10.02061062596699]
  Recent action-chunking methods have shown promising performance by predicting temporally extended action sequences.<n>We propose Uncertainty-Guided Refinement (UGR), a sparse refinement framework for chunk-based visuomotor policies.
  arXiv  Detail & Related papers  (2026-09-14T16:37:35Z)
• Future-as-Label: Scalable Supervision from Real-World Outcomes [0.0]
  Time creates free supervision: forecasts about real-world events resolve to verifiable outcomes.<n>We extend reinforcement learning with verifiable rewards to real-world prediction over time.<n>We train language models to make probabilistic forecasts from causally masked information.
  arXiv  Detail & Related papers  (2026-01-09T22:15:12Z)
• Reverberation: Learning the Latencies Before Forecasting Trajectories [18.978852734171593]
  Bridging the past to the future, connecting agents both spatially and temporally, lies at the core of the trajectory prediction task.<n>Different agents may exhibit distinct latency preferences for noticing, processing, and reacting to any specific trajectory-changing event.<n>Inspired by the reverberation curves in acoustics, we propose a new reverberation and the corresponding Reverberation trajectory prediction model.
  arXiv  Detail & Related papers  (2025-11-14T10:59:24Z)
• ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting [7.401111319849394]
  ForeSight is a novel joint detection and forecasting framework for vision-based 3D perception in autonomous vehicles.<n>We show that ForeSight achieves state-of-the-art performance, achieving an EPA of 54.9%, surpassing previous methods by 9.3%.
  arXiv  Detail & Related papers  (2025-08-09T20:18:10Z)
• Robust Planning for Autonomous Driving via Mixed Adversarial Diffusion Predictions [51.88828388720111]
  We describe a robust planning method for autonomous driving that mixes normal and adversarial agent predictions.<n>We generate a distribution of adversarial predictions by biasing the diffusion model at test time to generate predictions that are likely to collide with a candidate plan.<n>We show the effectiveness of our method on single-agent and multi-agent jaywalking scenarios as well as a red light violation scenario.
  arXiv  Detail & Related papers  (2025-05-18T09:44:57Z)
• Learning Through Retrospection: Improving Trajectory Prediction for Automated Driving with Error Feedback [41.94295877935867]
  In automated driving, predicting trajectories of surrounding vehicles supports reasoning about scene dynamics and enables safe planning for the ego vehicle.<n>Existing models handle predictions as an instantaneous task of forecasting future trajectories based on observed information.<n>We propose a novel retrospection technique to correct its errors during inference and will repeat them.
  arXiv  Detail & Related papers  (2025-04-18T16:35:12Z)
• Price predictability in limit order book with deep learning model [0.0]
  This study explores the prediction of high-frequency price changes using deep learning models. We found that an inadequately defined target price process may render predictions meaningless by incorporating past information.
  arXiv  Detail & Related papers  (2024-09-21T14:40:13Z)
• Movement-Prediction-Adjusted Naive Forecast [6.935130578959931]
  The movement-prediction-adjusted naive forecast (MPANF) is designed to improve point forecasts beyond the naive baseline.<n> MPANF can serve as an effective second-stage method when reliable movement predictions are available.
  arXiv  Detail & Related papers  (2024-06-20T16:32:18Z)
• Adaptive Human Trajectory Prediction via Latent Corridors [49.13061580045407]
  We formalize the problem of scene-specific adaptive trajectory prediction. We propose a new adaptation approach inspired by prompt tuning called latent corridors. With less than 0.1% additional model parameters, we see up to 23.9% ADE improvement in MOT Synth simulated data and 16.4% ADE in MOT and Wildtrack real pedestrian data.
  arXiv  Detail & Related papers  (2023-12-11T18:59:12Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• AdvDO: Realistic Adversarial Attacks for Trajectory Prediction [87.96767885419423]
  Trajectory prediction is essential for autonomous vehicles to plan correct and safe driving behaviors. We devise an optimization-based adversarial attack framework to generate realistic adversarial trajectories. Our attack can lead an AV to drive off road or collide into other vehicles in simulation.
  arXiv  Detail & Related papers  (2022-09-19T03:34:59Z)
• You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction [52.442129609979794]
  Recent deep learning approaches for trajectory prediction show promising performance. It remains unclear which features such black-box models actually learn to use for making predictions. This paper proposes a procedure that quantifies the contributions of different cues to model performance.
  arXiv  Detail & Related papers  (2021-10-11T14:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.