COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking

• URL: http://arxiv.org/abs/2605.14795v1
• Date: Thu, 14 May 2026 13:06:00 GMT
• Title: COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking
• Abstract summary: COAL (Counterfactual and Observation-enhanced Alignment Learning) is a framework that advances RMOT beyond isolated structural optimization through knowledge regularization.<n>We introduce Explicit Semantic Injection (ESI) via a VLM to densify the observation space and enhance instance discriminability.<n>We also propose Counterfactual Learning (CFL) to augment supervision, enforcing strict attribute verification for robust compositional recognition.
• Score: 38.34677413728821
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Multi-Object Tracking (RMOT) faces a fundamental structural contradiction between the high-discriminability demand and the sparse semantic supervision. This mismatch is particularly acute in highly homogeneous scenarios that require fine-grained discrimination over complex compositional semantics. However, under sparse supervision, models overfit to salient yet insufficient cues, thereby encouraging shortcut learning and semantic collapse. To resolve this, we propose COAL (Counterfactual and Observation-enhanced Alignment Learning), a framework that advances RMOT beyond isolated structural optimization through knowledge regularization. First, we introduce Explicit Semantic Injection (ESI) via a VLM to densify the observation space and enhance instance discriminability. Second, leveraging LLM reasoning, we propose Counterfactual Learning (CFL) to augment supervision, enforcing strict attribute verification for robust compositional recognition. These strategies are unified within a Hierarchical Multi-Stream Integration (HMSI) architecture, which distills external knowledge into domain-specific discriminative representations. Experiments on Refer-KITTI and Refer-KITTI-V2 benchmarks validate COAL's efficacy. Notably, it surpasses the state-of-the-art by 7.28% HOTA on the highly challenging Refer-KITTI-V2. These results demonstrate the effectiveness of knowledge regularization for resolving the sparsity-discriminability paradox in RMOT.

Related papers

• CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentation [15.230353995992246]
  Cross is a tightly integrated paradigm for referring Remote Sensing Image.<n>First, we introduce Linguistic-Guided Cascaded Distillation (LGCD) to bridge the architectural gap.<n>Second, Perspective-Spatial Contrastive Learning (PSCL) imposes cross-anchored constraints.
  arXiv  Detail & Related papers  (2026-08-04T05:24:05Z)
• Causal Bootstrapped Alignment for Unsupervised Video-Based Visible-Infrared Person Re-Identification [52.784239635604735]
  VVI-ReID is a critical technique for all-day surveillance, where temporal information provides additional cues beyond static images.<n>Existing approaches rely heavily on fully supervised learning with expensive cross-modality annotations, limiting scalability.<n>We propose a Causal Bootstrapped Alignment framework that explicitly exploits inherent video priors.
  arXiv  Detail & Related papers  (2026-04-17T02:15:59Z)
• Measure-Theoretic Anti-Causal Representation Learning [29.12751904333385]
  Causal representation learning in the anti-causal setting (labels cause features rather than the reverse) presents unique challenges.<n>We propose Anti-Causal Invariant Abstractions (ACIA), a novel measure-theoretic framework for anti-causal representation learning.<n>ACIA employs a two-level design, low-level representations capture how labels generate observations, while high-level representations learn stable causal patterns across environment-specific variations.
  arXiv  Detail & Related papers  (2025-10-16T22:13:05Z)
• Understanding Catastrophic Interference: On the Identifibility of Latent Representations [67.05452287233122]
  Catastrophic interference, also known as catastrophic forgetting, is a fundamental challenge in machine learning.<n>We propose a novel theoretical framework that formulates catastrophic interference as an identification problem.<n>Our approach provides both theoretical guarantees and practical performance improvements across both synthetic and benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-27T00:53:32Z)
• On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning [5.123232962822044]
  asymmetric non-contrastive learning (ANCL) often outperforms its contrastive learning counterpart in self-supervised representation learning. We study ANCL for supervised representation learning, coined SupSiam and SupBYOL, leveraging labels in ANCL to achieve better representations. Our analysis reveals that providing supervision to ANCL reduces intra-class variance, and the contribution of supervision should be adjusted to achieve the best performance.
  arXiv  Detail & Related papers  (2024-06-16T06:43:15Z)
• Learning Prompt-Enhanced Context Features for Weakly-Supervised Video Anomaly Detection [37.99031842449251]
  Video anomaly detection under weak supervision presents significant challenges. We present a weakly supervised anomaly detection framework that focuses on efficient context modeling and enhanced semantic discriminability. Our approach significantly improves the detection accuracy of certain anomaly sub-classes, underscoring its practical value and efficacy.
  arXiv  Detail & Related papers  (2023-06-26T06:45:16Z)
• Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems [61.11799513362704]
  We propose learning an additional screening mechanism to identify discriminative clues commonly seen across instances and classes. We show that a common rationale detector can be learned by simply exploiting the GradCAM induced from the SSL objective.
  arXiv  Detail & Related papers  (2023-03-03T02:07:40Z)
• Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness [69.39073806630583]
  Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields. We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
  arXiv  Detail & Related papers  (2022-07-22T06:30:44Z)
• Deep Clustering by Semantic Contrastive Learning [67.28140787010447]
  We introduce a novel variant called Semantic Contrastive Learning (SCL) It explores the characteristics of both conventional contrastive learning and deep clustering. It can amplify the strengths of contrastive learning and deep clustering in a unified approach.
  arXiv  Detail & Related papers  (2021-03-03T20:20:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.