Learning Hybrid Interpretable Models: Theory, Taxonomy, and Methods

• URL: http://arxiv.org/abs/2303.04437v1
• Date: Wed, 8 Mar 2023 08:35:29 GMT
• Title: Learning Hybrid Interpretable Models: Theory, Taxonomy, and Methods
• Abstract summary: A hybrid model involves the cooperation of an interpretable model and a complex black box. The advantages of such models over classical ones are two-fold. They grant users precise control over the level of transparency of the system.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A hybrid model involves the cooperation of an interpretable model and a complex black box. At inference, any input of the hybrid model is assigned to either its interpretable or complex component based on a gating mechanism. The advantages of such models over classical ones are two-fold: 1) They grant users precise control over the level of transparency of the system and 2) They can potentially perform better than a standalone black box since redirecting some of the inputs to an interpretable model implicitly acts as regularization. Still, despite their high potential, hybrid models remain under-studied in the interpretability/explainability literature. In this paper, we remedy this fact by presenting a thorough investigation of such models from three perspectives: Theory, Taxonomy, and Methods. First, we explore the theory behind the generalization of hybrid models from the Probably-Approximately-Correct (PAC) perspective. A consequence of our PAC guarantee is the existence of a sweet spot for the optimal transparency of the system. When such a sweet spot is attained, a hybrid model can potentially perform better than a standalone black box. Secondly, we provide a general taxonomy for the different ways of training hybrid models: the Post-Black-Box and Pre-Black-Box paradigms. These approaches differ in the order in which the interpretable and complex components are trained. We show where the state-of-the-art hybrid models Hybrid-Rule-Set and Companion-Rule-List fall in this taxonomy. Thirdly, we implement the two paradigms in a single method: HybridCORELS, which extends the CORELS algorithm to hybrid modeling. By leveraging CORELS, HybridCORELS provides a certificate of optimality of its interpretable component and precise control over transparency. We finally show empirically that HybridCORELS is competitive with existing hybrid models, and performs just as well as a standalone black box (or even better) while being partly transparent.

Related papers

• When Interpretability Is Unequally Distributed: Fairness in Hybrid Interpretable Models [7.412725493312841]
  Hybrid interpretable models combine a transparent component with a black-box model by assigning some examples to the former and deferring the rest to the latter.<n>Some demographic groups may systematically receive interpretable decisions, while others are disproportionately routed to a black box.<n>We formalize this issue as Interpretability Coverage Disparity (ICD), a demographic-parity-style measure applied to the routing decision of hybrid interpretable models.
  arXiv  Detail & Related papers  (2026-05-27T15:33:00Z)
• Olmo Hybrid: From Theory to Practice and Back [112.70006263561073]
  We show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both.<n>We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations.<n>Our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm.
  arXiv  Detail & Related papers  (2026-04-03T20:36:34Z)
• Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models [50.45915413315706]
  We prove the existence of fundamental limitations for non-hybrid models.<n>We construct hybrid models of small size and working memory that provably solve these tasks.<n>We additionally demonstrate that hybrid models exhibit stronger length generalization and out-of-distribution robustness than non-hybrids.
  arXiv  Detail & Related papers  (2026-03-09T19:20:01Z)
• UNBOX: Unveiling Black-box visual models with Natural-language [50.433977345786055]
  We introduce UNBOX, a framework for class-wise model dissection under fully data-free, gradient-free, and backpropagation-free constraints.<n>We evaluate UNBOX on ImageNet-1K, Waterbirds, and CelebA through semantic fidelity tests, visual-feature correlation analyses, and slice-discovery auditing.
  arXiv  Detail & Related papers  (2026-03-09T17:16:39Z)
• Learning Deep Hybrid Models with Sharpness-Aware Minimization [4.8941886361557625]
  We propose to focus on the flatness of loss minima in learning hybrid models, aiming to make the model as simple as possible.<n>We employ the idea of sharpness-aware minimization and adapt it to the hybrid modeling setting.<n> Numerical experiments show that the SAM-based method works well across different choices of models and datasets.
  arXiv  Detail & Related papers  (2026-02-06T16:27:19Z)
• BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning [84.56022893225422]
  Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge.<n>Applying vision-language models such as CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities.
  arXiv  Detail & Related papers  (2025-11-14T15:51:40Z)
• Gnothi Seauton: Empowering Faithful Self-Interpretability in Black-Box Models [21.698201509643624]
  Self-interpretable models, such as concept-based networks, offer insights by connecting decisions to human-understandable concepts. Post-hoc methods like Shapley values, while theoretically robust, are computationally expensive and resource-intensive. We propose a novel method that combines their strengths, providing theoretically guaranteed self-interpretability for black-box models.
  arXiv  Detail & Related papers  (2024-10-29T07:35:33Z)
• Exploring Model Kinship for Merging Large Language Models [52.01652098827454]
  We introduce model kinship, the degree of similarity or relatedness between Large Language Models. We find that there is a certain relationship between model kinship and the performance gains after model merging. We propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can yield better performance on benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-16T14:29:29Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models [54.132297393662654]
  We introduce a hybrid method that fine-tunes cutting-edge diffusion models by optimizing reward models through RL. We demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models.
  arXiv  Detail & Related papers  (2024-05-30T03:57:29Z)
• Hybrid Quantum-inspired Resnet and Densenet for Pattern Recognition [1.0499611180329804]
  We propose two hybrid quantum-inspired neural networks with adaptive residual and dense connections respectively for pattern recognition. We show the potential superiority of our hybrid models to prevent gradient explosion owing to the quantum-inspired layers.
  arXiv  Detail & Related papers  (2024-03-09T01:34:26Z)
• Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation [71.21346469382821]
  We introduce collaborative black-box tuning (CBBT) for both textual prompt optimization and output feature adaptation for black-box models. CBBT is extensively evaluated on eleven downstream benchmarks and achieves remarkable improvements compared to existing black-box VL adaptation methods.
  arXiv  Detail & Related papers  (2023-12-26T06:31:28Z)
• Quality Diversity Evolutionary Learning of Decision Trees [4.447467536572625]
  We show that MAP-Elites can diversify hybrid models over a feature space that captures both the model complexity and its behavioral variability. We apply our method on two well-known control problems from the OpenAI Gym library, on which we discuss the "illumination" patterns projected by MAP-Elites.
  arXiv  Detail & Related papers  (2022-08-17T13:57:32Z)
• Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization [6.204805504959941]
  We propose a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. Our innovations, including dynamic online kernel selection in the surrogate modeling phase, position our hybrid models as an advancement in mixed-variable surrogate models.
  arXiv  Detail & Related papers  (2022-06-03T06:34:09Z)
• Robust Hybrid Learning With Expert Augmentation [31.911717646180886]
  We introduce a hybrid data augmentation strategy termed textitexpert augmentation We demonstrate that expert augmentation, which can be incorporated into existing hybrid systems, improves generalization. We also assess the potential real-world applicability of expert augmentation on a dataset of a real double pendulum.
  arXiv  Detail & Related papers  (2022-02-08T14:11:04Z)
• Have best of both worlds: two-pass hybrid and E2E cascading framework for speech recognition [71.30167252138048]
  Hybrid and end-to-end (E2E) systems have different error patterns in the speech recognition results. This paper proposes a two-pass hybrid and E2E cascading (HEC) framework to combine the hybrid and E2E model. We show that the proposed system achieves 8-10% relative word error rate reduction with respect to each individual system.
  arXiv  Detail & Related papers  (2021-10-10T20:11:38Z)
• Self-Supervised Hybrid Inference in State-Space Models [0.0]
  We perform approximate inference in state-space models that allow for nonlinear higher-order Markov chains in latent space. We do not rely on an additional parameterization of the generative model or supervision via uncorrupted observations or ground truth latent states. We obtain competitive results on the chaotic Lorenz system compared to a fully supervised approach and outperform a method based on variational inference.
  arXiv  Detail & Related papers  (2021-07-28T13:26:14Z)
• UnitedQA: A Hybrid Approach for Open Domain Question Answering [70.54286377610953]
  We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively.
  arXiv  Detail & Related papers  (2021-01-01T06:36:16Z)
• Reconstruction of Pairwise Interactions using Energy-Based Models [3.553493344868414]
  We show that hybrid models, which combine a pairwise model and a neural network, can lead to significant improvements in the reconstruction of pairwise interactions. This is in line with the general idea that simple interpretable models and complex black-box models are not necessarily a dichotomy.
  arXiv  Detail & Related papers  (2020-12-11T20:15:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.