Augmentations for Robust and Efficient Imitation Learning in Streamed Video Games

• URL: http://arxiv.org/abs/2607.14200v1
• Date: Wed, 15 Jul 2026 17:07:51 GMT
• Title: Augmentations for Robust and Efficient Imitation Learning in Streamed Video Games
• Abstract summary: We propose streaming augmentations that mimic four types of artifacts commonly encountered during streaming with low-bandwidth network connection.<n>Under stable streaming conditions, identicaltemporal augmentations achieve up to 41% higher evaluation performance compared to agents trained without augmentations.<n>When network lag is introduced, agents trained with augmentations degrade by only 7.45% vs 49.82% of the original performance for agents trained only with the original data.
• Score: 8.793009090321926
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Imitation learning is an appealing way to scale game-playing agents to complex 3D environments by training policies to map visual observations to actions from human demonstrations. However, these demonstrations are expensive to collect and modern game-playing is often done through streaming in which network delay and compression introduce spatiotemporally correlated visual artifacts that can cause a covariance shift at test time. To address these challenges, we propose streaming augmentations that mimic four types of artifacts commonly encountered during streaming with low-bandwidth network connection: pixelated blocks and scrubs, global blur, and ghosting. We instantiate our approach on top of predictive inverse dynamics models (PIDM), which combine future-state conditioning with an inverse dynamics policy in a learned latent space, and evaluate the impact of our augmentations across three tasks in modern 3D video games. Under stable streaming conditions, agents trained with spatiotemporal augmentations achieve up to 41% higher evaluation performance compared to agents trained without augmentations under an identical data budget. When network lag is introduced, agents trained with augmentations degrade by only 7.45% vs 49.82% of the original performance for agents trained only with the original data. These results clearly indicate that spatiotemporal augmentations tailored for the streaming setting are a simple yet powerful tool to train robust and efficient game-playing agents.

Related papers

• COMBAT: Conditional World Models for Behavioral Agent Training [2.7205188669026277]
  We introduce COMBAT, a real-time, action-controlled world model trained on the complex 1v1 fighting game Tekken 3.<n>Our work demonstrates that diffusion models can successfully simulate a dynamic opponent that reacts to player actions.<n>We observe the emergence of sophisticated agent behavior by training the model solely on single-player inputs.
  arXiv  Detail & Related papers  (2026-02-28T22:09:04Z)
• Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents [56.25101378553328]
  We present Game-TARS, a generalist game agent trained with a unified, scalable action space anchored to human-aligned keyboard-mouse inputs.<n>Game-TARS is pre-trained on over 500B tokens with diverse trajectories and multimodal data.<n> Experiments show that Game-TARS achieves about 2 times the success rate over the previous sota model on open-world Minecraft tasks.
  arXiv  Detail & Related papers  (2025-10-27T17:43:51Z)
• Learning Representations in Video Game Agents with Supervised Contrastive Imitation Learning [0.6299766708197881]
  This paper introduces a novel application of Supervised Contrastive Learning (SupCon) to Imitation Learning (IL)<n>The goal is to obtain latent representations of the observations that capture better the action-relevant factors.<n> Experiments on the 3D games Astro Bot and Returnal, and multiple 2D Atari games show improved representation quality, faster learning convergence, and better generalization.
  arXiv  Detail & Related papers  (2025-09-15T13:00:29Z)
• MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning [66.53533434848369]
  We propose a motion-guided self-learning framework that learns densely consistent representations.<n>We improve state-of-the-art by 1% to 6% on six image and video datasets and four evaluation benchmarks.
  arXiv  Detail & Related papers  (2025-06-10T11:20:32Z)
• DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control [18.737628473949048]
  Imitation learning has proven to be a powerful tool for training complex visuomotor policies. Current methods often require hundreds to thousands of expert demonstrations to handle high-dimensional visual observations. We present DynaMo, a new in-domain, self-supervised method for learning visual representations.
  arXiv  Detail & Related papers  (2024-09-18T17:59:43Z)
• Real-time Object Detection for Streaming Perception [84.2559631820007]
  Streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. We build a simple and effective framework for streaming perception. Our method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline.
  arXiv  Detail & Related papers  (2022-03-23T11:33:27Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)
• Spatiotemporal Contrastive Video Representation Learning [87.56145031149869]
  We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn visual representations from unlabeled videos. Our representations are learned using a contrasttemporalive loss, where two augmented clips from the same short video are pulled together in the embedding space. We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial.
  arXiv  Detail & Related papers  (2020-08-09T19:58:45Z)
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations [55.41644538483948]
  In this paper, we propose a combination of approaches that allow the agent to use low-quality demonstrations in complex vision-based environments. Our proposed goal-oriented structuring of replay buffer allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. The solution based on our algorithm beats all the solutions for the famous MineRL competition and allows the agent to mine a diamond in the Minecraft environment.
  arXiv  Detail & Related papers  (2020-06-17T15:38:40Z)
• Towards Learning to Imitate from a Single Video Demonstration [11.15358253586118]
  We develop a reinforcement learning agent that can learn to imitate given video observation. We use a Siamese recurrent neural network architecture to learn rewards in space and time between motion clips. We demonstrate our approach on simulated humanoid, dog, and raptor agents in 2D and a quadruped and a humanoid in 3D.
  arXiv  Detail & Related papers  (2019-01-22T06:46:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.