SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM

• URL: http://arxiv.org/abs/2603.08269v1
• Date: Mon, 09 Mar 2026 11:39:40 GMT
• Title: SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM
• Abstract summary: In-context imitation learning allows robots to acquire skills from demonstrations, yet one-shot trajectory generation remains fragile under environmental variation.<n>We propose a framework that reframes robot imitation as an iterative refinement problem capable of scaling with test-time compute.
• Score: 24.04193602666315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context imitation learning allows robots to acquire skills from demonstrations, yet one-shot trajectory generation remains fragile under environmental variation. We propose SAIL, a framework that reframes robot imitation as an iterative refinement problem capable of scaling with test-time compute. SAIL utilizes Monte Carlo Tree Search, where each node is a complete trajectory and edges correspond to trajectory refinements. The process is guided by three core components: an automated archive of successful trajectories for contextually relevant retrieval, a vision language model-based scoring mechanism for trajectory evaluation, and a step-level feedback that provides trajectory-aligned scores for iterative refinement. Experiments across six diverse manipulation tasks in simulation and real-world validation clearly demonstrate that increasing test-time compute consistently improves success rates, achieving up to 95% on complex tasks. Our results suggest that trajectory-level test-time scaling is a robust path toward more generalizable robotic agents.

Related papers

• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons [69.87766750714945]
  General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations.<n>We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision.<n>Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints.
  arXiv  Detail & Related papers  (2026-03-02T17:38:58Z)
• Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation [54.945281159783896]
  We present a scalable pipeline for automatically generating high-quality training data for web agents.<n>We introduce a novel constraint-based evaluation framework that provides fine-grained assessment of progress towards task completion.
  arXiv  Detail & Related papers  (2026-02-13T02:52:18Z)
• Optimizing Control-Friendly Trajectories with Self-Supervised Residual Learning [10.551597725130675]
  Real-world physics can only be analytically modeled with a certain level of precision for modern intricate robotic systems.<n>This paper presents a self-supervised residual learning and trajectory optimization framework to address the aforementioned challenges.
  arXiv  Detail & Related papers  (2026-01-06T06:01:58Z)
• VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Models [49.78447737655287]
  VITA is a zero-shot value function learning method that enhances both capabilities via test-time adaptation.<n>We demonstrate that VITA's zero-shot value estimates can be utilized for reward shaping in offline reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-11T18:05:33Z)
• Action Flow Matching for Continual Robot Learning [54.10050120844738]
  Continual learning in robotics seeks systems that can constantly adapt to changing environments and tasks.<n>We introduce a generative framework leveraging flow matching for online robot dynamics model alignment.<n>We find that by transforming the actions themselves rather than exploring with a misaligned model, the robot collects informative data more efficiently.
  arXiv  Detail & Related papers  (2025-04-25T16:26:15Z)
• STeCa: Step-level Trajectory Calibration for LLM Agent Learning [9.678098205818566]
  Large language model (LLM)-based agents have shown promise in tackling complex tasks by interacting dynamically with the environment.<n>We propose Step-Level Trajectory (STeCa), a novel framework for LLM agent learning.
  arXiv  Detail & Related papers  (2025-02-20T05:28:44Z)
• A Backbone for Long-Horizon Robot Task Understanding [8.889888977376886]
  Therblig-Based Backbone Framework (TBBF) is a structure to enhance interpretability, data efficiency, and generalization in robotic systems.<n>TBBF utilizes expert demonstrations to enable therblig-level task decomposition.<n>During the offline training stage, we developed the Meta-RGate SynerFusion network for accurate therblig segmentation.<n>In the online testing stage, after a one-shot demonstration of a new task is collected, our MGSF network extracts high-level knowledge.
  arXiv  Detail & Related papers  (2024-08-02T15:32:42Z)
• Self-Supervised Representation Learning from Temporal Ordering of Automated Driving Sequences [49.91741677556553]
  We propose TempO, a temporal ordering pretext task for pre-training region-level feature representations for perception tasks. We embed each frame by an unordered set of proposal feature vectors, a representation that is natural for object detection or tracking systems. Extensive evaluations on the BDD100K, nuImages, and MOT17 datasets show that our TempO pre-training approach outperforms single-frame self-supervised learning methods.
  arXiv  Detail & Related papers  (2023-02-17T18:18:27Z)
• Deep Learning Traversability Estimator for Mobile Robots in Unstructured Environments [11.042142015353626]
  We propose a deep learning framework, trained in an end-to-end fashion from elevation maps and trajectories, to estimate the occurrence of failure events. We show that transferring and fine-tuning of an application-independent pre-trained model retains better performance than training uniquely on scarcely available real data.
  arXiv  Detail & Related papers  (2021-05-23T13:49:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.