Robot Critics that Sweat the Small Stuff
Abstract Overview
This paper studies how to make vision-language model critics useful for closed-loop robot manipulation, where success and failure can differ by very small visual details such as slight misalignment or missed contact. The authors fine-tune a critic using pairwise supervision built from matched successful and failed policy rollouts, so the model learns both incremental task progress and policy-specific failure modes. At test time, the critic is placed in the loop with a stochastic policy: multiple action candidates are sampled, an action-conditioned video model predicts their visual outcomes, and the critic ranks the candidates by comparative progress. Experiments on RoboCasa, the real-world LBM benchmark, tabletop robot tasks, and RoboCasa365 simulation indicate that this approach improves fine-grained progress discrimination and can steer policies toward better action choices.
Novelty
The main novelty is a critic-training scheme that uses matched success and failure rollouts from the same initial conditions to create pairwise progress supervision targeted at subtle, policy-specific failures. The work also combines this fine-tuned critic with action-conditioned video prediction for test-time reranking of sampled action trajectories in closed-loop robot control.
Results
The fine-tuned critic substantially outperforms prompted and off-the-shelf baselines on fine-grained progress/failure judgment, reaching 0.936 average accuracy on RoboCasa and 0.900 average accuracy on the real-world LBM dataset, with strong OOD performance. In closed-loop use, the method improves average policy success by 11% across real-world tasks and 5.9% across simulation tasks; examples include Stacking improving from 23/50 to 32/50 trials and Push-Bowl mIoU increasing from 0.140 to 0.321.
Key Points
- Training on both successful and failed rollouts is important: a success-only critic drops markedly on success-failure comparisons, while the full method better recognizes subtle failure modes.
- The critic is trained as a pairwise comparator of predicted terminal states, which is better aligned with choosing among candidate policy rollouts than assigning absolute progress scores.
- Performance gains depend partly on the action-conditioned video model, and the paper notes failure cases from prediction errors and substantial inference latency for video synthesis.