Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations

• URL: http://arxiv.org/abs/2607.04235v1
• Date: Sun, 05 Jul 2026 11:21:26 GMT
• Title: Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations
• Abstract summary: We introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-language goals the agent actually achieved.<n>HSL pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning.<n>Our experiments show that HSL is flexible and compatible with existing post-training pipelines.
• Score: 53.931465193216155
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language model agents operate in partially observable, long-horizon settings where obtaining supervision remains a major bottleneck. We address this by utilizing a source of supervision overlooked in existing post-training methods: unintended yet successful goals embedded within agent rollouts. Specifically, we introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-language goals the agent actually achieved. HSL then pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning. To mitigate suboptimality in the relabeled data, we propose two learning techniques for HSL, irrelevant-action masking and sample reweighting. Our experiments show that HSL is flexible and compatible with existing post-training pipelines. It improves both SFT and DPO, with larger gains on long-horizon tasks with more diverse goal spaces. Moreover, HSL is sample-efficient: on ALFWorld, it surpasses baselines trained on the full dataset while using only one quarter of the ground-truth demonstrations.

Related papers

• Memory-Based Advantage Shaping for LLM-Guided Reinforcement Learning [18.215893951726166]
  In environments with sparse or delayed rewards, reinforcement learning incurs high sample complexity.<n>This limitation has motivated the use of large language models (LLMs) for subgoal discovery and trajectory guidance.<n>We address these challenges by constructing a memory graph that encodes subgoals and trajectories from both LLM guidance and the agent's own successful rollouts.
  arXiv  Detail & Related papers  (2026-02-20T01:44:35Z)
• Language-based Trial and Error Falls Behind in the Era of Experience [50.503828360874536]
  Large Language Models (LLMs) excel in language-based agentic tasks, but their applicability to unseen, nonlinguistic environments remains limited.<n>In this work, we demonstrate the primary bottleneck is the prohibitive cost of exploration.<n>We propose SCOUT, a novel framework that decouples exploration from semantic exploitation.
  arXiv  Detail & Related papers  (2026-01-29T14:08:41Z)
• Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective [27.35951269350466]
  Semi-supervised few-shot learning formulates real-world applications like ''auto-annotation''<n>We propose Stage-Wise Finetuning with Temperature Tuning (SWIFT), which enables existing SSL methods to effectively finetune a VLM on limited labeled data, abundant unlabeled data, and task-relevant but noisy data retrieved from the VLM's pretraining set.
  arXiv  Detail & Related papers  (2025-12-11T03:06:16Z)
• Self-Improving Vision-Language-Action Models with Data Generation via Residual RL [29.682761652941963]
  Probe, Learn, Distill (PLD) is a three-stage plug-and-play framework that improves vision-language-action models.<n>PLD achieves near-saturated 99% task success on LIBERO, over 50% gains in SimplerEnv, and 100% success on real-world Franka and YAM arm manipulation tasks.
  arXiv  Detail & Related papers  (2025-10-30T06:24:04Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)
• STeCa: Step-level Trajectory Calibration for LLM Agent Learning [9.678098205818566]
  Large language model (LLM)-based agents have shown promise in tackling complex tasks by interacting dynamically with the environment.<n>We propose Step-Level Trajectory (STeCa), a novel framework for LLM agent learning.
  arXiv  Detail & Related papers  (2025-02-20T05:28:44Z)
• SEAL: SEmantic-Augmented Imitation Learning via Language Model [16.345410506033915]
  SEAL is a novel framework for specifying sub-goal space and pre-labeling states to semantically meaningful sub-goal representations. SEAL employs a dual-encoder structure, combining supervised LLM-guided sub-goal learning with unsupervised Vector Quantization. Our experiments demonstrate that SEAL outperforms state-of-the-art HIL methods and LLM-based planning approaches.
  arXiv  Detail & Related papers  (2024-10-03T05:53:10Z)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems [59.40480894948944]
  Large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. We prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning.
  arXiv  Detail & Related papers  (2024-05-30T09:42:54Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.