Sim-to-Real Causal Transfer: A Metric Learning Approach to Causally-Aware Interaction Representations

• URL: http://arxiv.org/abs/2312.04540v1
• Date: Thu, 7 Dec 2023 18:57:03 GMT
• Title: Sim-to-Real Causal Transfer: A Metric Learning Approach to Causally-Aware Interaction Representations
• Authors: Yuejiang Liu, Ahmad Rahimi, Po-Chien Luan, Frano Raji\v{c}, Alexandre Alahi
• Abstract summary: We take an in-depth look at the causal awareness of modern representations of agent interactions. We show that recent representations are already partially resilient to perturbations of non-causal agents. We propose a metric learning approach that regularizes latent representations with causal annotations.
• Score: 62.48505112245388
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modeling spatial-temporal interactions among neighboring agents is at the heart of multi-agent problems such as motion forecasting and crowd navigation. Despite notable progress, it remains unclear to which extent modern representations can capture the causal relationships behind agent interactions. In this work, we take an in-depth look at the causal awareness of these representations, from computational formalism to real-world practice. First, we cast doubt on the notion of non-causal robustness studied in the recent CausalAgents benchmark. We show that recent representations are already partially resilient to perturbations of non-causal agents, and yet modeling indirect causal effects involving mediator agents remains challenging. To address this challenge, we introduce a metric learning approach that regularizes latent representations with causal annotations. Our controlled experiments show that this approach not only leads to higher degrees of causal awareness but also yields stronger out-of-distribution robustness. To further operationalize it in practice, we propose a sim-to-real causal transfer method via cross-domain multi-task learning. Experiments on pedestrian datasets show that our method can substantially boost generalization, even in the absence of real-world causal annotations. We hope our work provides a new perspective on the challenges and potential pathways towards causally-aware representations of multi-agent interactions. Our code is available at https://github.com/socialcausality.

Related papers

• A Survey on Transferability of Adversarial Examples across Deep Neural Networks [53.04734042366312]
  adversarial examples can manipulate machine learning models into making erroneous predictions. The transferability of adversarial examples enables black-box attacks which circumvent the need for detailed knowledge of the target model. This survey explores the landscape of the adversarial transferability of adversarial examples.
  arXiv  Detail & Related papers  (2023-10-26T17:45:26Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• Neural Amortized Inference for Nested Multi-agent Reasoning [54.39127942041582]
  We propose a novel approach to bridge the gap between human-like inference capabilities and computational limitations. We evaluate our method in two challenging multi-agent interaction domains.
  arXiv  Detail & Related papers  (2023-08-21T22:40:36Z)
• Why Does Little Robustness Help? Understanding and Improving Adversarial Transferability from Surrogate Training [24.376314203167016]
  Adversarial examples (AEs) for DNNs have been shown to be transferable. In this paper, we take a further step towards understanding adversarial transferability.
  arXiv  Detail & Related papers  (2023-07-15T19:20:49Z)
• Simulation-Based Counterfactual Causal Discovery on Real World Driver Behaviour [6.273003557777915]
  We present three variants of the proposed counterfactual causal discovery method. We evaluate these against state of the art observational temporal causal discovery methods across 3396 causal scenes extracted from a real world driving dataset.
  arXiv  Detail & Related papers  (2023-06-06T02:10:20Z)
• Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning [98.78136504619539]
  Causal Triplet is a causal representation learning benchmark featuring visually more complex scenes. We show that models built with the knowledge of disentangled or object-centric representations significantly outperform their distributed counterparts.
  arXiv  Detail & Related papers  (2023-01-12T17:43:38Z)
• Causal Reinforcement Learning using Observational and Interventional Data [14.856472820492364]
  Learning efficiently a causal model of the environment is a key challenge of model RL agents operating in POMDPs. We consider a scenario where the learning agent has the ability to collect online experiences through direct interactions with the environment. We then ask the following questions: can the online and offline experiences be safely combined for learning a causal model.
  arXiv  Detail & Related papers  (2021-06-28T06:58:20Z)
• Feedback in Imitation Learning: Confusion on Causality and Covariate Shift [12.93527098342393]
  We argue that conditioning policies on previous actions leads to a dramatic divergence between "held out" error and performance of the learner in situ. We analyze existing benchmarks used to test imitation learning approaches. We find, in a surprising contrast with previous literature, that naive behavioral cloning provides excellent results.
  arXiv  Detail & Related papers  (2021-02-04T20:18:56Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.