CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.22846v2
• Date: Tue, 31 Mar 2026 09:31:35 GMT
• Title: CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models
• Abstract summary: Embodied Visual Tracking (EVT) is a core dynamic task in embodied intelligence.<n>We propose CoMaTrack, a competitive game-theoretic multi-agent reinforcement learning framework.<n>We introduce CoMaTrack-Bench, the first open-source benchmark protocol and episode set for language-conditioned competitive EVT.
• Score: 17.223855925754837
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied Visual Tracking (EVT), a core dynamic task in embodied intelligence, requires an agent to precisely follow a language-specified target. Yet most existing methods rely on single-agent imitation learning, suffering from costly expert data and limited generalization due to static training environments. Inspired by competition-driven capability evolution, we propose CoMaTrack, a competitive game-theoretic multi-agent reinforcement learning framework that trains agents in a dynamic adversarial setting with competitive subtasks, yielding stronger adaptive planning and interference-resilient strategies. We further introduce CoMaTrack-Bench, the first open-source Habitat-based benchmark protocol and episode set for language-conditioned competitive EVT featuring dynamic dueling, featuring game scenarios between a tracker and adaptive opponents across diverse environments and instructions, enabling standardized robustness evaluation under active adversarial interactions. Experiments show that CoMaTrack achieves state-of-the-art results on both standard benchmarks and CoMaTrack-Bench. Notably, a 3B VLM trained with our framework surpasses previous single-agent imitation learning methods based on 7B models on the challenging EVT-Bench, achieving 92.1% in STT, 74.2% in DT, and 57.5% in AT. The benchmark code will be available at https://github.com/wlqcode/CoMaTrack-Bench.

Related papers

• ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search [54.43696074323815]
  ZGCM-1 is a fully open 7B foundation model trained from scratch with extreme data, system, and algorithmic efficiency.<n>We establish an AI-native R&D workflow where agent swarms autonomously manage cluster operations.<n>ZGCM-1-7B is competitive across 7B model family on general benchmarks.
  arXiv  Detail & Related papers  (2026-09-11T17:18:04Z)
• Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation [60.96010213186819]
  We introduce Interactive Tracking, a new paradigm that allows users to guide the tracker at any time using natural language commands.<n>We present InteractTrack, the first large-scale benchmark for interactive tracking, containing 150 videos with dense bounding box annotations and timestamped language instructions.<n>Third, we introduce Interactive Memory-Augmented Tracking (IMAT), a new baseline that employs a dynamic memory mechanism to learn from user feedback and update tracking behavior accordingly.
  arXiv  Detail & Related papers  (2026-04-02T12:33:27Z)
• DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI [84.9072161615971]
  We present DM0, an Embodied-Native Vision-Language-Action framework designed for Physical AI.<n>Our methodology follows a comprehensive three-stage pipeline: Pretraining, Mid-Training, and Post-Training.<n>Experiments on the RoboChallenge benchmark demonstrate that DM0 achieves state-of-the-art performance in both Specialist and Generalist settings.
  arXiv  Detail & Related papers  (2026-02-16T17:59:16Z)
• TrackVLA: Embodied Visual Tracking in the Wild [34.03604806748204]
  Embodied visual tracking is a fundamental skill in Embodied AI, enabling an agent to follow a specific target in dynamic environments using only egocentric vision.<n>Existing approaches typically address this challenge through a modular separation of recognition and planning.<n>We propose TrackVLA, a Vision-Language-Action model that learns the synergy between object recognition and trajectory planning.
  arXiv  Detail & Related papers  (2025-05-29T07:28:09Z)
• Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark [72.46357004059661]
  Generalist Virtual Agents (GVAs) have shown significant promise in autonomous task execution.<n>To address these challenges, we propose Similar, a Step-Wise Multi-Dimensional Generalist Reward Model.<n>Similar offers fine-grained signals for agent training and can choose better action for inference-time scaling.
  arXiv  Detail & Related papers  (2025-03-24T13:30:47Z)
• SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking [34.90147791481045]
  SynCL is a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking.<n>We propose a Task-specific Hybrid Matching module for a weight-shared cross-attention-based decoder.<n>We also introduce Instance-aware Contrastive Learning to break through the barrier of self-centric attention for track queries.
  arXiv  Detail & Related papers  (2024-11-11T08:18:49Z)
• A Minimaximalist Approach to Reinforcement Learning from Human Feedback [49.45285664482369]
  We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training. We demonstrate that on a suite of continuous control tasks, we are able to learn significantly more efficiently than reward-model based approaches.
  arXiv  Detail & Related papers  (2024-01-08T17:55:02Z)
• MixFormer: End-to-End Tracking with Iterative Mixed Attention [47.78513247048846]
  We present a compact tracking framework, termed as MixFormer, built upon transformers. We propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. Our MixFormer trackers set a new state-of-the-art performance on seven tracking benchmarks.
  arXiv  Detail & Related papers  (2023-02-06T14:38:09Z)
• Learn to Match: Automatic Matching Network Design for Visual Tracking [43.476030483114094]
  We introduce six novel matching operators instead of explicit similarity learning. We propose binary channel manipulation to search for the optimal combination of these operators. Our model achieves favorable gains by $67.2 rightarrow 71.4$, $52.6 rightarrow 58.3$, $70.3 rightarrow 76.0$ success on OTB100, LaSOT, and TrackingNet.
  arXiv  Detail & Related papers  (2021-08-02T12:09:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.