CounterAlign: Counterfactual Supervision for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2608.21740v1
• Date: Sat, 22 Aug 2026 02:43:34 GMT
• Title: CounterAlign: Counterfactual Supervision for Vision-Language-Action Models
• Abstract summary: Reinforcement learning can provide such corrective signals, but often relies on externally specified rewards or curated non-expert data.<n>We show that offline RL for VLA models can be transformed into dense corrective supervision through instruction relabeling.<n>Our method improves robustness to object position and task perturbations over a strong state-of-the-art baseline.
• Score: 11.229017941425278
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models are typically trained with behavior cloning (BC) on expert demonstrations. However, BC provides only positive supervision for expert actions, without explicit negative supervision indicating which actions are instruction-inconsistent or otherwise inappropriate. Reinforcement learning (RL) can provide such corrective signals, but often relies on externally specified rewards or curated non-expert data, both of which are costly to obtain in robotics. We show that offline RL for VLA models need not rely on curated non-expert trajectories: successful expert demonstrations alone can be transformed into dense corrective supervision through instruction relabeling. Specifically, by pairing expert actions with mismatched alternative instructions, we synthesize counterfactual instruction-observation-action tuples from the dataset and combine them with adversarial discriminator training to learn an instruction-grounded reward model for offline RL, without collecting additional rollouts or annotations. On the robustness-focused LIBERO-PRO benchmark, our method improves robustness to object position and task perturbations over a strong state-of-the-art baseline. It also outperforms competitive baselines in real-robot experiments on the TX-G2 (compatible with AGIBot G2). More broadly, our results suggest that, for data-constrained VLA learning, extracting denser supervision from each demonstration can complement collecting additional data.

Related papers

• PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents [57.69188221227875]
  We propose a Privileged trAce Co-Training framework for multi-turn tool-use agents.<n>The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints.<n>We show that PACT consistently improves over strong SFT- and RL-based baselines.
  arXiv  Detail & Related papers  (2026-06-15T04:46:23Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following [58.60470643433354]
  Language models often struggle to follow multi-constraint instructions that are crucial for real-world applications.<n>We propose a label-free self-supervised reinforcement learning framework that eliminates dependency on external supervision.<n>Our approach introduces constraint decomposition strategies and efficient constraint-wise binary classification to address sparse reward challenges.
  arXiv  Detail & Related papers  (2025-10-16T08:24:44Z)
• Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations [10.679604514849744]
  We study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations.<n>We propose a novel formulation that optimize a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data.<n>Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework.
  arXiv  Detail & Related papers  (2025-05-27T13:33:21Z)
• SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data [65.56911325914582]
  We propose Self-play Reinforcement Learning (SeRL) to bootstrap Large Language Models (LLMs) training with limited initial data.<n>The proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards.
  arXiv  Detail & Related papers  (2025-05-25T13:28:04Z)
• PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement [16.768912344111946]
  We present PROGRESSOR, a framework that learns a task-agnostic reward function from videos.<n>We show that PROGRESSOR enables robots to learn complex behaviors without any external supervision.
  arXiv  Detail & Related papers  (2024-11-26T04:17:51Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• Language Models Are Poor Learners of Directional Inference [17.807086499130488]
  LMs show limited ability to learn such directional inference. Existing datasets fail to test directionality. Existing LM-prompting models are incompetent directional entailment learners.
  arXiv  Detail & Related papers  (2022-10-10T13:43:16Z)
• Imitation by Predicting Observations [17.86983397979034]
  We present a new method for imitation solely from observations that achieves comparable performance to experts on challenging continuous control tasks. Our method, which we call FORM, is derived from an inverse RL objective and imitates using a model of expert behavior learned by generative modelling of the expert's observations. We show that FORM performs comparably to a strong baseline IRL method (GAIL) on the DeepMind Control Suite benchmark, while outperforming GAIL in the presence of task-irrelevant features.
  arXiv  Detail & Related papers  (2021-07-08T14:09:30Z)
• Offline Learning from Demonstrations and Unlabeled Experience [62.928404936397335]
  Behavior Imitation (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. This unlabeled data can be generated by a variety of sources such as human teleoperation, scripted policies and other agents on the same robot. We show that Offline Reinforced Learning (ORIL) consistently outperforms comparable BC agents by effectively leveraging unlabeled experience.
  arXiv  Detail & Related papers  (2020-11-27T18:20:04Z)
• Robust Disentanglement of a Few Factors at a Time [5.156484100374058]
  We introduce population-based training (PBT) for improving consistency in training variational autoencoders (VAEs) We then use Unsupervised Disentanglement Ranking (UDR) as an unsupervised to score models in our PBT-VAE training and show how models trained this way tend to consistently disentangle only a subset of the generative factors. We show striking improvement in state-of-the-art unsupervised disentanglement performance and robustness across multiple datasets and metrics.
  arXiv  Detail & Related papers  (2020-10-26T12:34:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.