Contextual Mixture of Experts: Integrating Knowledge into Predictive Modeling

• URL: http://arxiv.org/abs/2211.00558v1
• Date: Tue, 1 Nov 2022 16:12:42 GMT
• Title: Contextual Mixture of Experts: Integrating Knowledge into Predictive Modeling
• Authors: Francisco Souza, Tim Offermans, Ruud Barendse, Geert Postma, Jeroen Jansen
• Abstract summary: This work proposes a new data-driven model devised to integrate process knowledge into its structure to increase the human-machine synergy in the process industry. The proposed Contextual Mixture of Experts (cMoE) explicitly uses process knowledge along the model learning stage to mold the historical data to represent operators' context related to the process through possibility distributions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This work proposes a new data-driven model devised to integrate process knowledge into its structure to increase the human-machine synergy in the process industry. The proposed Contextual Mixture of Experts (cMoE) explicitly uses process knowledge along the model learning stage to mold the historical data to represent operators' context related to the process through possibility distributions. This model was evaluated in two real case studies for quality prediction, including a sulfur recovery unit and a polymerization process. The contextual mixture of experts was employed to represent different contexts in both experiments. The results indicate that integrating process knowledge has increased predictive performance while improving interpretability by providing insights into the variables affecting the process's different regimes.

Related papers

• Large Language Models for Extrapolative Modeling of Manufacturing Processes [5.705795836910535]
  The novelty lies in combining automatic extraction of process-relevant knowledge embedded in the literature with iterative model refinement based on a small amount of experimental data. The results show that for the same small experimental data budget the models derived by our framework have unexpectedly high extrapolative performance.
  arXiv  Detail & Related papers  (2025-02-15T02:43:22Z)
• Fast Training Dataset Attribution via In-Context Learning [9.542023122304096]
  We investigate the use of in-context learning and prompt engineering to estimate the contributions of training data in instruction-tuned large language models (LLMs) We propose two novel approaches: (1) a similarity-based approach that measures the difference between LLM outputs with and without provided context, and (2) a mixture distribution model approach that frames the problem of identifying contribution scores as a matrix factorization task.
  arXiv  Detail & Related papers  (2024-08-14T20:48:45Z)
• Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts [75.85448576746373]
  We propose a method of grouping and pruning similar experts to improve the model's parameter efficiency. We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures. The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks.
  arXiv  Detail & Related papers  (2024-07-12T17:25:02Z)
• Interpretable and Explainable Machine Learning Methods for Predictive Process Monitoring: A Systematic Literature Review [1.3812010983144802]
  This paper presents a systematic review on the explainability and interpretability of machine learning (ML) models within the context of predictive process mining. We provide a comprehensive overview of the current methodologies and their applications across various application domains. Our findings aim to equip researchers and practitioners with a deeper understanding of how to develop and implement more trustworthy, transparent, and effective intelligent systems for process analytics.
  arXiv  Detail & Related papers  (2023-12-29T12:43:43Z)
• Contextualized Machine Learning [40.415518395978204]
  Contextualized Machine Learning estimates heterogeneous functions by applying deep learning to the meta-relationship between contextual information and context-specific parametric models. We present the open-source PyTorch package ContextualizedML.
  arXiv  Detail & Related papers  (2023-10-17T15:23:00Z)
• Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis [89.04041100520881]
  This research proposes to retrieve textual and visual evidence based on the object, sentence, and whole image. We develop a novel approach to synthesize the object-level, image-level, and sentence-level information for better reasoning between the same and different modalities.
  arXiv  Detail & Related papers  (2023-05-25T15:26:13Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• Extending Process Discovery with Model Complexity Optimization and Cyclic States Identification: Application to Healthcare Processes [62.997667081978825]
  The paper presents an approach to process mining providing semi-automatic support to model optimization. A model simplification approach is proposed, which essentially abstracts the raw model at the desired granularity. We aim to demonstrate the capabilities of the technological solution using three datasets from different applications in the healthcare domain.
  arXiv  Detail & Related papers  (2022-06-10T16:20:59Z)
• KAT: A Knowledge Augmented Transformer for Vision-and-Language [56.716531169609915]
  We propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. An additional benefit of explicit knowledge integration is seen in improved interpretability of model predictions in our analysis.
  arXiv  Detail & Related papers  (2021-12-16T04:37:10Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Local Post-Hoc Explanations for Predictive Process Monitoring in Manufacturing [0.0]
  This study proposes an innovative explainable predictive quality analytics solution to facilitate data-driven decision-making in manufacturing. It combines process mining, machine learning, and explainable artificial intelligence (XAI) methods.
  arXiv  Detail & Related papers  (2020-09-22T13:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.