A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning

• URL: http://arxiv.org/abs/2608.24195v1
• Date: Tue, 25 Aug 2026 08:00:40 GMT
• Title: A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning
• Abstract summary: Mixture-of-Experts (MoE) models provide a flexible framework for partitioning complex prediction problems into simpler local learning tasks.<n>Existing interpretable MoE approaches, such as Mixture of Decision Trees (MoDT), achieve transparency by employing homogeneous decision-tree experts.<n>We extend the MoDT framework by introducing heterogeneous expert families comprising decision trees, linear support vector machines, and quadratic discriminant analysis.
• Score: 0.30586855806896035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) models provide a flexible framework for partitioning complex prediction problems into simpler local learning tasks through an input-dependent gating mechanism. Existing interpretable MoE approaches, such as Mixture of Decision Trees (MoDT), achieve transparency by employing homogeneous decision-tree experts, but this restricts the model to a single inductive bias across all regions of the feature space. We extend the MoDT framework by introducing heterogeneous expert families comprising decision trees, linear support vector machines, and quadratic discriminant analysis under a common probabilistic gating mechanism. To ensure coherent likelihood-based inference, non-probabilistic experts are calibrated to produce conditional class probabilities, allowing parameter estimation within the generalized Expectation-Maximization framework of MoDT. We further establish theoretical monotone ascent guarantees for the proposed heterogeneous gating updates, providing a justification for the optimization procedure. Experiments on a diverse collection of synthetic and real-world benchmark datasets demonstrate that the proposed framework adaptively specializes experts according to local data geometry, yielding interpretable expert assignments while achieving predictive performance competitive with homogeneous MoDT and Random Forests. The proposed approach combines interpretability, adaptive inductive bias selection, and probabilistic coherence within a unified mixture-of-experts framework.

Related papers

• AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE [51.994752158953084]
  AnchorMoE is an interpretable-by-construction classification framework.<n>It encodes multi-view representations of local patches and routes them to specialized experts.<n>Experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance.
  arXiv  Detail & Related papers  (2026-06-02T13:30:54Z)
• On Bayesian Softmax-Gated Mixture-of-Experts Models [95.06359956519759]
  We study mixture-of-experts models, focusing on the ubiquitous softmax-based gating mechanism.<n>We analyze parameter estimation and derive convergence guarantees based on tailored Voronoi-type losses.<n> Taken together, these results provide one of the first systematic theoretical analyses of mixture-of-experts models with softmax gating.
  arXiv  Detail & Related papers  (2026-04-22T13:37:00Z)
• ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding [23.27523995613204]
  We propose ABLE (Attribution-Based Large-model Embedding), a framework that leverages the interpretability space to construct model representations.<n>By aggregating gradient-based feature attributions via a tokenizer-agnostic word-level alignment, ABLE captures model-specific input-sensitivity patterns.<n>Our training-free approach achieves competitive or superior performance in relation prediction, model routing, and benchmark score prediction.
  arXiv  Detail & Related papers  (2026-04-20T04:55:11Z)
• Gaussian Joint Embeddings For Self-Supervised Representation Learning [0.0]
  Self-parametric representation learning often relies on deterministic predictive architectures to align context and target views in latent space.<n>We propose a probabilistic alternative based on generative joint modeling.
  arXiv  Detail & Related papers  (2026-03-26T00:54:54Z)
• Mixture-of-experts Wishart model for covariance matrices with an application to Cancer drug screening [0.8594140167290097]
  We introduce a comprehensive Bayesian framework for analyzing heterogeneous covariance data through both classical mixture models and a novel mixture-of-experts Wishart model.<n>We develop an efficient Gibbs-within-Metropolis-Hastings sampling algorithm tailored to the geometry of the Wishart likelihood and the gating network.<n>We present an innovative application of our methodology to a challenging dataset: cancer drug sensitivity profiles.
  arXiv  Detail & Related papers  (2026-02-14T21:07:14Z)
• Varying-Coefficient Mixture of Experts Model [0.0]
  We propose a Varying-Coefficient Mixture of Experts (VCMoE) model that allows all coefficient effects in both the gating functions and expert models to vary along an indexing variable.<n>We illustrate the proposed VCMoE model using a dataset of single gene expression in embryonic mice to characterize the temporal dynamics of the associations between the expression levels of genes Satb2 and Bcl11b.
  arXiv  Detail & Related papers  (2026-01-05T00:23:30Z)
• Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning [0.0]
  Hecto is a lightweight MoE architecture that combines a GRU expert for temporal reasoning and an FFNN expert for static abstraction under a sparse Top-1 gating mechanism.<n>Hecto matches or closely trails homogeneous baselines in performance despite receiving isolated input representations.<n>Hecto establishes itself as a new benchmark for conditional computation.
  arXiv  Detail & Related papers  (2025-06-28T15:03:43Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)
• Pseudo-Spherical Contrastive Divergence [119.28384561517292]
  We propose pseudo-spherical contrastive divergence (PS-CD) to generalize maximum learning likelihood of energy-based models. PS-CD avoids the intractable partition function and provides a generalized family of learning objectives.
  arXiv  Detail & Related papers  (2021-11-01T09:17:15Z)
• Identification of Probability weighted ARX models with arbitrary domains [75.91002178647165]
  PieceWise Affine models guarantees universal approximation, local linearity and equivalence to other classes of hybrid system. In this work, we focus on the identification of PieceWise Auto Regressive with eXogenous input models with arbitrary regions (NPWARX) The architecture is conceived following the Mixture of Expert concept, developed within the machine learning field.
  arXiv  Detail & Related papers  (2020-09-29T12:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.