CausalMoE: A Billion-Scale Multimodal Foundation Model for Granger Causal Discovery with Pattern-Routed Heterogeneous Experts

• URL: http://arxiv.org/abs/2606.13024v1
• Date: Thu, 11 Jun 2026 07:57:23 GMT
• Title: CausalMoE: A Billion-Scale Multimodal Foundation Model for Granger Causal Discovery with Pattern-Routed Heterogeneous Experts
• Abstract summary: We propose CausalMoE, a billion-scale multimodal Granger causal foundation model.<n>CaulMoE introduces a Pattern-Routed Mixture of Heterogeneous Experts, which dynamically identifies latent temporal patterns.<n>We also design a Causality-Aware Self-Attention mechanism operating across variables, yielding sparse Granger causal graphs.
• Score: 25.563603697914242
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Granger Causal Discovery (GCD) is fundamental for analyzing temporal dependencies in complex systems. However, existing neural GCD methods predominantly rely on a "one-size-fits-all" paradigm, struggling to capture distribution shifts and dynamic regime changes inherent in real-world time series. This often leads to entangled representations and spurious causal graphs. In this paper, we propose CausalMoE, a billion-scale multimodal Granger causal foundation model that explicitly models patch-level heterogeneity. CausalMoE introduces a Pattern-Routed Mixture of Heterogeneous Experts, which dynamically identifies latent temporal patterns and routes patches to specialized domain experts, effectively decoupling regime-specific mechanisms from shared dynamics. To ensure interpretable graph recovery, we design a Causality-Aware Self-Attention mechanism operating across variables, yielding sparse Granger causal graphs via proximal optimization. Furthermore, CausalMoE is the first to integrate LLMs and VLMs to align numerical signals with textual and visual priors, regularizing causal estimation in complex scenarios. Extensive experiments demonstrate that CausalMoE establishes a new state-of-the-art on fully supervised benchmarks, while effectively generalizing to few-shot settings where traditional methods fail.

Related papers

• DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation [23.80343378702536]
  Multimodal Retrieval-Augmented Generation (MM-RAG) has shown promising results, but it still struggles with complex multi-hop reasoning tasks.<n>We propose DualG-MRAG, a Dual-tier framework that introduces a decoupled architecture comprising Macro-reasoning and Micro-matching Graphs for Multimodal RAG.
  arXiv  Detail & Related papers  (2026-07-30T17:40:05Z)
• DAG-FM: A Foundation Model for Causal Discovery under Heterogeneous Causal Mechanisms [48.57846236686172]
  We propose textbfDAG-FM, a novel foundation model architecture that amortizes causal discovery.<n>Unlike direct matrix prediction, DAG-FM decomposes the causal discovery process into two auto-regressive stages.<n>We show that DAG-FM achieves state-of-the-art performance on both synthetic benchmarks and complex real-world datasets.
  arXiv  Detail & Related papers  (2026-07-13T13:00:06Z)
• CDFM: Towards a General-Purpose Causal Discovery Foundation Model [62.32138962508216]
  Causal discovery is the process of recovering underlying causal structures from observational data.<n>We formulate the Causal Discovery Foundation Model (CDFM) as a unified, general-purpose framework for zero-shot structural inference.<n>CDFM consistently outperforms traditional algorithms, driving a paradigm shift toward a general-purpose causal discovery foundation model.
  arXiv  Detail & Related papers  (2026-07-13T12:58:28Z)
• Time Series Causal Discovery via Context-Conditioned and Causality-Augmented Pretraining [17.01926647313838]
  Causal discovery from time series is critical for many real-world applications, such as tracing the root causes of anomalies.<n>We propose textbfPTCD, a novel textbfPretraining framework for textbfTime-series textbfCausal textbfDiscovery.
  arXiv  Detail & Related papers  (2026-05-26T09:30:42Z)
• Evolutionary Router Feature Generation for Zero-Shot Graph Anomaly Detection with Mixture-of-Experts [60.60414602796664]
  We propose a novel MoE framework with evolutionary router feature generation (EvoFG) for zero-shot GAD.<n>EvoFG consistently outperforms state-of-the-art baselines, achieving strong and stable zero-shot GAD performance.
  arXiv  Detail & Related papers  (2026-02-12T06:16:51Z)
• Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains [50.66049136093248]
  We develop a time-aware structural causal model (SCM) that incorporates dynamic causal factors and the causal mechanism drifts.<n>We show that our method can yield the optimal causal predictor for each time domain.<n>Results on both synthetic and real-world datasets exhibit that SYNC can achieve superior temporal generalization performance.
  arXiv  Detail & Related papers  (2025-06-21T14:05:37Z)
• Multi-View Causal Discovery without Non-Gaussianity: Identifiability and Algorithms [59.15672758767244]
  Causal discovery is a difficult problem that typically relies on strong assumptions on the data-generating model, such as non-Gaussianity.<n>Here, we leverage this multi-view structure to achieve causal discovery with weak assumptions.<n>We propose several multi-view causal discovery algorithms, inspired by single-view algorithms.
  arXiv  Detail & Related papers  (2025-02-27T14:06:14Z)
• Jacobian Regularizer-based Neural Granger Causality [45.902407376192656]
  We propose a Jacobian Regularizer-based Neural Granger Causality (JRNGC) approach. Our method eliminates the sparsity constraints of weights by leveraging an input-output Jacobian matrix regularizer. Our proposed approach achieves competitive performance with the state-of-the-art methods for learning summary Granger causality and full-time Granger causality.
  arXiv  Detail & Related papers  (2024-05-14T17:13:50Z)
• Causal Temporal Regime Structure Learning [49.77103348208835]
  We present CASTOR, a novel method that concurrently learns the Directed Acyclic Graph (DAG) for each regime.<n>We establish the identifiability of the regimes and DAGs within our framework.<n>Experiments show that CASTOR consistently outperforms existing causal discovery models.
  arXiv  Detail & Related papers  (2023-11-02T17:26:49Z)
• Inductive Granger Causal Modeling for Multivariate Time Series [49.29373497269468]
  We propose an Inductive GRanger cAusal modeling (InGRA) framework for inductive Granger causality learning and common causal structure detection. In particular, we train one global model for individuals with different Granger causal structures through a novel attention mechanism, called Granger causal attention. The model can detect common causal structures for different individuals and infer Granger causal structures for newly arrived individuals.
  arXiv  Detail & Related papers  (2021-02-10T07:48:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.