MOCA: A Transformer-based Modular Causal Inference Framework with One-way Cross-attention and Cutting Feedback

• URL: http://arxiv.org/abs/2604.23107v1
• Date: Sat, 25 Apr 2026 02:05:49 GMT
• Title: MOCA: A Transformer-based Modular Causal Inference Framework with One-way Cross-attention and Cutting Feedback
• Abstract summary: We propose MOCA, a transformer-based framework that separates treatment and outcome modeling through a modular design.<n>A cutting-feedback strategy, implemented via gradient detachment, prevents the outcome loss from updating the treatment module.<n>Across multiple simulated scenarios, MOCA shows competitive or improved performance relative to IPW, AIPW, X-learner, TARNet, and DragonNet.
• Score: 7.234620259018605
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Causal effect estimation from observational data requires careful adjustment for confounding. Classical estimators such as inverse probability weighting and augmented inverse probability weighting are effective under favorable model specification, but may become unstable when treatment assignment and outcome mechanisms are complex, non-linear, and high-dimensional. Machine learning and representation learning approaches improve flexibility, yet joint training can allow outcome-related information to influence treatment-side representations, which is undesirable from a causal perspective. We propose MOCA (Modular One-way Causal Attention), a transformer-based framework that separates treatment and outcome modeling through a modular design, and performs confounder adjustment using a one-way attention mechanism. A cutting-feedback strategy, implemented via gradient detachment, prevents the outcome loss from updating the treatment module. This design preserves directional information flow while retaining the representational power of transformer architectures for causal inference. Across multiple simulated scenarios, including linear, nonlinear, heavy-tailed, hidden confounding, and high-dimensional settings, MOCA shows competitive or improved performance relative to IPW, AIPW, X-learner, TARNet, and DragonNet. We further illustrate the method on the Infant Health and Development Program dataset and the Dehejia-Wahba dataset as real-world benchmarks. These results suggest that modular attention with one-way information flow provides a promising and interpretable direction for causal inference with modern deep learning models.

Related papers

• Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal [29.196846051811576]
  We introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations.<n>We show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality.
  arXiv  Detail & Related papers  (2026-06-10T17:31:16Z)
• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• Guided Verifier: Collaborative Multimodal Reasoning via Dynamic Process Supervision [11.159231524113764]
  Reinforcement Learning (RL) has emerged as a pivotal mechanism for enhancing the complex reasoning capabilities of Multimodal Large Language Models (MLLMs)<n>In this paper, we propose the textbfGuided Verifier framework to address these structural limitations.<n>We develop a specialized data synthesis pipeline targeting multimodal hallucinations, constructing textbfCoRe dataset of process-level negatives and textbfCorrect-guide textbfReasoning trajectories to train the guided verifier.
  arXiv  Detail & Related papers  (2026-02-04T07:38:42Z)
• Learning Causality for Longitudinal Data [1.2691047660244335]
  This thesis develops methods for causal inference and causal representation learning in high-dimensional, time-varying data.<n>The first contribution introduces the Causal Dynamic Variational Autoencoder (CDVAE), a model for estimating Individual Treatment Effects (ITEs)<n>The second contribution proposes an efficient framework for long-term counterfactual regression based on RNNs enhanced with Contrastive Predictive Coding ( CPC) and InfoMax.<n>The third contribution advances CRL by addressing how latent causes manifest in observed variables.
  arXiv  Detail & Related papers  (2025-12-04T16:51:49Z)
• Causal Model-Based Reinforcement Learning for Sample-Efficient IoT Channel Access [39.76683291751265]
  Multi-agent reinforcement learning (MARL) for wireless use case such as medium access control (MAC) is hindered by their sample inefficiency.<n>In this paper, a novel causal model-based MARL framework is developed by leveraging tools from causal learn- ing.<n>The proposed approach can reduce environment interactions by 58%, and yield faster convergence compared to model-free baselines.
  arXiv  Detail & Related papers  (2025-11-13T13:26:33Z)
• Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency [4.047219770183742]
  Time series forecasting plays a pivotal role in critical domains such as energy management and financial markets.<n>This study reveals a counterintuitive phenomenon: appropriately truncating historical data can enhance prediction accuracy.<n>We propose an innovative solution termed Adaptive Masking Loss with Representation Consistency.
  arXiv  Detail & Related papers  (2025-10-22T19:23:53Z)
• From Physics to Machine Learning and Back: Part II - Learning and Observational Bias in PHM [52.64097278841485]
  Review examines how incorporating learning and observational biases through physics-informed modeling and data strategies can guide models toward physically consistent and reliable predictions.<n>Fast adaptation methods including meta-learning and few-shot learning are reviewed alongside domain generalization techniques.
  arXiv  Detail & Related papers  (2025-09-25T14:15:43Z)
• Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation [62.14692332209628]
  "Interaction Distillation" is a novel training framework for more adequate preference modeling through attention-level optimization.<n>It provides more stable and generalizable reward signals compared to state-of-the-art RM optimization methods.
  arXiv  Detail & Related papers  (2025-08-04T17:06:23Z)
• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models [4.737806982257592]
  This study proposes a knowledge distillation algorithm based on large language models and feature alignment.<n>The proposed model performs very close to the state-of-the-art GPT-4 model in terms of evaluation indicators such as perplexity, BLEU, ROUGE, and CER.
  arXiv  Detail & Related papers  (2024-12-27T04:37:06Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling [66.3072381478251]
  Reward hacking, also termed reward overoptimization, remains a critical challenge. We propose a framework for reward modeling, namely InfoRM, by introducing a variational information bottleneck objective. We show that InfoRM's overoptimization detection mechanism is not only effective but also robust across a broad range of datasets.
  arXiv  Detail & Related papers  (2024-02-14T17:49:07Z)
• Over-the-Air Federated Learning and Optimization [52.5188988624998]
  We focus on Federated learning (FL) via edge-the-air computation (AirComp) We describe the convergence of AirComp-based FedAvg (AirFedAvg) algorithms under both convex and non- convex settings. For different types of local updates that can be transmitted by edge devices (i.e., model, gradient, model difference), we reveal that transmitting in AirFedAvg may cause an aggregation error. In addition, we consider more practical signal processing schemes to improve the communication efficiency and extend the convergence analysis to different forms of model aggregation error caused by these signal processing schemes.
  arXiv  Detail & Related papers  (2023-10-16T05:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.