Towards Robust Knowledge Tracing Models via k-Sparse Attention
- URL: http://arxiv.org/abs/2407.17097v1
- Date: Wed, 24 Jul 2024 08:49:18 GMT
- Title: Towards Robust Knowledge Tracing Models via k-Sparse Attention
- Authors: Shuyan Huang, Zitao Liu, Xiangyu Zhao, Weiqi Luo, Jian Weng,
- Abstract summary: textscsparseKT is a simple yet effective framework to improve the robustness and generalization of the attention based DLKT approaches.
We show that our textscsparseKT is able to help attentional KT models get rid of irrelevant student interactions.
- Score: 33.02197868261949
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Knowledge tracing (KT) is the problem of predicting students' future performance based on their historical interaction sequences. With the advanced capability of capturing contextual long-term dependency, attention mechanism becomes one of the essential components in many deep learning based KT (DLKT) models. In spite of the impressive performance achieved by these attentional DLKT models, many of them are often vulnerable to run the risk of overfitting, especially on small-scale educational datasets. Therefore, in this paper, we propose \textsc{sparseKT}, a simple yet effective framework to improve the robustness and generalization of the attention based DLKT approaches. Specifically, we incorporate a k-selection module to only pick items with the highest attention scores. We propose two sparsification heuristics : (1) soft-thresholding sparse attention and (2) top-$K$ sparse attention. We show that our \textsc{sparseKT} is able to help attentional KT models get rid of irrelevant student interactions and have comparable predictive performance when compared to 11 state-of-the-art KT models on three publicly available real-world educational datasets. To encourage reproducible research, we make our data and code publicly available at \url{https://github.com/pykt-team/pykt-toolkit}\footnote{We merged our model to the \textsc{pyKT} benchmark at \url{https://pykt.org/}.}.
Related papers
- Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing [59.480951050911436]
We present KCQRL, a framework for automated knowledge concept annotation and question representation learning.
We demonstrate the effectiveness of KCQRL across 15 KT algorithms on two large real-world Math learning datasets.
arXiv Detail & Related papers (2024-10-02T16:37:19Z) - Improving Low-Resource Knowledge Tracing Tasks by Supervised Pre-training and Importance Mechanism Fine-tuning [25.566963415155325]
We propose a low-resource KT framework called LoReKT to address above challenges.
Inspired by the prevalent "pre-training and fine-tuning" paradigm, we aim to learn transferable parameters and representations from rich-resource KT datasets.
We design an encoding mechanism to incorporate student interactions from multiple KT data sources.
arXiv Detail & Related papers (2024-03-11T13:44:43Z) - Retrieval-Enhanced Contrastive Vision-Text Models [61.783728119255365]
We propose to equip vision-text models with the ability to refine their embedding with cross-modal retrieved information from a memory at inference time.
Remarkably, we show that this can be done with a light-weight, single-layer, fusion transformer on top of a frozen CLIP.
Our experiments validate that our retrieval-enhanced contrastive (RECO) training improves CLIP performance substantially on several challenging fine-grained tasks.
arXiv Detail & Related papers (2023-06-12T15:52:02Z) - simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing [22.055683237994696]
We provide a strong but simple baseline method to deal with the KT task named textscsimpleKT.
Inspired by the Rasch model in psychometrics, we explicitly model question-specific variations to capture the individual differences among questions.
We use the ordinary dot-product attention function to extract the time-aware information embedded in the student learning interactions.
arXiv Detail & Related papers (2023-02-14T08:09:09Z) - Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based
Action Recognition [88.34182299496074]
Action labels are only available on a source dataset, but unavailable on a target dataset in the training stage.
We utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets.
By segmenting and permuting temporal segments or human body parts, we design two self-supervised learning classification tasks.
arXiv Detail & Related papers (2022-07-17T07:05:39Z) - pyKT: A Python Library to Benchmark Deep Learning based Knowledge
Tracing Models [46.05383477261115]
Knowledge tracing (KT) is the task of using students' historical learning interaction data to model their knowledge mastery over time.
DLKT approaches are still left somewhat unknown and proper measurement and analysis of these approaches remain a challenge.
We introduce a comprehensive python based benchmark platform, textscpyKT, to guarantee valid comparisons across DLKT methods.
arXiv Detail & Related papers (2022-06-23T02:42:47Z) - Enhancing Knowledge Tracing via Adversarial Training [5.461665809706664]
We study the problem of knowledge tracing (KT) where the goal is to trace the students' knowledge mastery over time.
Recent advances on KT have increasingly concentrated on exploring deep neural networks (DNNs) to improve the performance of KT.
We propose an efficient AT based KT method (ATKT) to enhance KT model's generalization and thus push the limit of KT.
arXiv Detail & Related papers (2021-08-10T03:35:13Z) - How Knowledge Graph and Attention Help? A Quantitative Analysis into
Bag-level Relation Extraction [66.09605613944201]
We quantitatively evaluate the effect of attention and Knowledge Graph on bag-level relation extraction (RE)
We find that (1) higher attention accuracy may lead to worse performance as it may harm the model's ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns; and (3) KG-enhanced attention indeed improves RE performance, while not through enhanced attention but by incorporating entity prior.
arXiv Detail & Related papers (2021-07-26T09:38:28Z) - qDKT: Question-centric Deep Knowledge Tracing [29.431121650577396]
We introduce qDKT, a variant of DKT that models every learner's success probability on individual questions over time.
qDKT incorporates graph Laplacian regularization to smooth predictions under each skill.
Experiments on several real-world datasets show that qDKT achieves state-of-art performance on predicting learner outcomes.
arXiv Detail & Related papers (2020-05-25T23:43:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.