MOOCRep: A Unified Pre-trained Embedding of MOOC Entities

• URL: http://arxiv.org/abs/2107.05154v1
• Date: Mon, 12 Jul 2021 00:11:25 GMT
• Title: MOOCRep: A Unified Pre-trained Embedding of MOOC Entities
• Authors: Shalini Pandey, Jaideep Srivastava
• Abstract summary: We propose to learn pre-trained representations of MOOC entities using abundant unlabeled data from the structure of MOOCs. Our experiments reveal that MOOCRep's embeddings outperform state-of-the-art representation learning methods on two tasks important for education community.
• Score: 4.0963355240233446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many machine learning models have been built to tackle information overload issues on Massive Open Online Courses (MOOC) platforms. These models rely on learning powerful representations of MOOC entities. However, they suffer from the problem of scarce expert label data. To overcome this problem, we propose to learn pre-trained representations of MOOC entities using abundant unlabeled data from the structure of MOOCs which can directly be applied to the downstream tasks. While existing pre-training methods have been successful in NLP areas as they learn powerful textual representation, their models do not leverage the richer information about MOOC entities. This richer information includes the graph relationship between the lectures, concepts, and courses along with the domain knowledge about the complexity of a concept. We develop MOOCRep, a novel method based on Transformer language model trained with two pre-training objectives : 1) graph-based objective to capture the powerful signal of entities and relations that exist in the graph, and 2) domain-oriented objective to effectively incorporate the complexity level of concepts. Our experiments reveal that MOOCRep's embeddings outperform state-of-the-art representation learning methods on two tasks important for education community, concept pre-requisite prediction and lecture recommendation.

Related papers

• Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning [79.46570165281084]
  We propose a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods. MulKI achieves this through four stages, including Eliciting Ideas, Adding New Ideas, Distinguishing Ideas, and Making Connections. Our method demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks.
  arXiv  Detail & Related papers  (2024-11-11T07:36:19Z)
• Modeling Balanced Explicit and Implicit Relations with Contrastive Learning for Knowledge Concept Recommendation in MOOCs [1.0377683220196874]
  Existing methods rely on the explicit relations between users and knowledge concepts for recommendation. There are numerous implicit relations generated within the users' learning activities on the MOOC platforms. We propose a novel framework based on contrastive learning, which can represent and balance the explicit and implicit relations.
  arXiv  Detail & Related papers  (2024-02-13T07:12:44Z)
• Text Representation Distillation via Information Bottleneck Principle [22.63996326177594]
  We propose a novel Knowledge Distillation method called IBKD. It aims to maximize the mutual information between the final representation of the teacher and student model, while simultaneously reducing the mutual information between the student model's representation and the input data. Empirical studies on two main downstream applications of text representation demonstrate the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2023-11-09T16:04:17Z)
• Learn From Model Beyond Fine-Tuning: A Survey [78.80920533793595]
  Learn From Model (LFM) focuses on the research, modification, and design of foundation models (FM) based on the model interface. The study of LFM techniques can be broadly categorized into five major areas: model tuning, model distillation, model reuse, meta learning and model editing. This paper gives a comprehensive review of the current methods based on FM from the perspective of LFM.
  arXiv  Detail & Related papers  (2023-10-12T10:20:36Z)
• Contextualized Structural Self-supervised Learning for Ontology Matching [0.9402105308876642]
  We introduce a novel self-supervised learning framework called LaKERMap. LaKERMap capitalizes on the contextual and structural information of concepts by integrating implicit knowledge into transformers. The findings from our innovative approach reveal that LaKERMap surpasses state-of-the-art systems in terms of alignment quality and inference time.
  arXiv  Detail & Related papers  (2023-10-05T18:51:33Z)
• MoCoSA: Momentum Contrast for Knowledge Graph Completion with Structure-Augmented Pre-trained Language Models [11.57782182864771]
  We propose Momentum Contrast for knowledge graph completion with Structure-Augmented pre-trained language models (MoCoSA) Our approach achieves state-of-the-art performance in terms of mean reciprocal rank (MRR), with improvements of 2.5% on WN18RR and 21% on OpenBG500.
  arXiv  Detail & Related papers  (2023-08-16T08:09:10Z)
• Learning without Forgetting for Vision-Language Models [65.49600786387106]
  Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world. Recent advances in Vision-Language Models (VLM) have shown promising capabilities in learning generalizable representations. We propose PROjectiOn Fusion (PROOF) that enables VLMs to learn without forgetting.
  arXiv  Detail & Related papers  (2023-05-30T17:59:32Z)
• Scaling Vision-Language Models with Sparse Mixture of Experts [128.0882767889029]
  We show that mixture-of-experts (MoE) techniques can achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling vision-language models.
  arXiv  Detail & Related papers  (2023-03-13T16:00:31Z)
• Multi-View MOOC Quality Evaluation via Information-Aware Graph Representation Learning [26.723385384507274]
  We develop an Information-aware Graph Representation Learning(IaGRL) for multi-view MOOC quality evaluation. We first build a MOOC Heterogeneous Network (HIN) to represent the interactions and relationships among entities in MOOC platforms. And then we decompose the MOOC HIN into multiple single-relation graphs based on meta-paths to depict the multi-view semantics of courses.
  arXiv  Detail & Related papers  (2023-01-01T10:27:06Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.