A Generalized Apprenticeship Learning Framework for Modeling Heterogeneous Student Pedagogical Strategies

• URL: http://arxiv.org/abs/2406.02450v1
• Date: Tue, 4 Jun 2024 16:14:55 GMT
• Title: A Generalized Apprenticeship Learning Framework for Modeling Heterogeneous Student Pedagogical Strategies
• Authors: Md Mirajul Islam, Xi Yang, John Hostetter, Adittya Soukarjya Saha, Min Chi,
• Abstract summary: We propose an expectation-maximization(EM)-EDM, a general AL framework to induce effective pedagogical policies from given optimal or near-optimal demonstrations. We compare the effectiveness of the policies induced by our proposed EM-EDM against four AL-based baselines and two policies induced by DRL.
• Score: 8.137664701386198
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A key challenge in e-learning environments like Intelligent Tutoring Systems (ITSs) is to induce effective pedagogical policies efficiently. While Deep Reinforcement Learning (DRL) often suffers from sample inefficiency and reward function design difficulty, Apprenticeship Learning(AL) algorithms can overcome them. However, most AL algorithms can not handle heterogeneity as they assume all demonstrations are generated with a homogeneous policy driven by a single reward function. Still, some AL algorithms which consider heterogeneity, often can not generalize to large continuous state space and only work with discrete states. In this paper, we propose an expectation-maximization(EM)-EDM, a general AL framework to induce effective pedagogical policies from given optimal or near-optimal demonstrations, which are assumed to be driven by heterogeneous reward functions. We compare the effectiveness of the policies induced by our proposed EM-EDM against four AL-based baselines and two policies induced by DRL on two different but related tasks that involve pedagogical action prediction. Our overall results showed that, for both tasks, EM-EDM outperforms the four AL baselines across all performance metrics and the two DRL baselines. This suggests that EM-EDM can effectively model complex student pedagogical decision-making processes through the ability to manage a large, continuous state space and adapt to handle diverse and heterogeneous reward functions with very few given demonstrations.

Related papers

• On-the-fly Modulation for Balanced Multimodal Learning [53.616094855778954]
  Multimodal learning is expected to boost model performance by integrating information from different modalities. The widely-used joint training strategy leads to imbalanced and under-optimized uni-modal representations. We propose On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies to modulate the optimization of each modality.
  arXiv  Detail & Related papers  (2024-10-15T13:15:50Z)
• Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques [65.55451717632317]
  We study Multi-Agent Reinforcement Learning from Human Feedback (MARLHF), exploring both theoretical foundations and empirical validations. We define the task as identifying Nash equilibrium from a preference-only offline dataset in general-sum games. Our findings underscore the multifaceted approach required for MARLHF, paving the way for effective preference-based multi-agent systems.
  arXiv  Detail & Related papers  (2024-09-01T13:14:41Z)
• Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning [54.40927310957792]
  We introduce a novel concept of personalized expert demonstrations, tailored for each individual agent or, more broadly, each individual type of agent within a heterogeneous team. These demonstrations solely pertain to single-agent behaviors and how each agent can achieve personal goals without encompassing any cooperative elements. We propose an approach that selectively utilizes personalized expert demonstrations as guidance and allows agents to learn to cooperate.
  arXiv  Detail & Related papers  (2024-03-13T20:11:20Z)
• Inverse Reinforcement Learning by Estimating Expertise of Demonstrators [18.50354748863624]
  IRLEED, Inverse Reinforcement Learning by Estimating Expertise of Demonstrators, is a novel framework that overcomes hurdles without prior knowledge of demonstrator expertise. IRLEED enhances existing Inverse Reinforcement Learning (IRL) algorithms by combining a general model for demonstrator suboptimality to address reward bias and action variance. Experiments in both online and offline IL settings, with simulated and human-generated data, demonstrate IRLEED's adaptability and effectiveness.
  arXiv  Detail & Related papers  (2024-02-02T20:21:09Z)
• Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations [22.23114883485924]
  We propose a novel algorithm called GENTLE for learning generalizable task representations in the face of data limitations. GENTLE employs Task Auto-Encoder(TAE), which is an encoder-decoder architecture to extract the characteristics of the tasks. To alleviate the effect of limited behavior diversity, we construct pseudo-transitions to align the data distribution used to train TAE with the data distribution encountered during testing.
  arXiv  Detail & Related papers  (2023-12-26T07:02:12Z)
• Task Phasing: Automated Curriculum Learning from Demonstrations [46.1680279122598]
  Applying reinforcement learning to sparse reward domains is notoriously challenging due to insufficient guiding signals. This paper introduces a principled task phasing approach that uses demonstrations to automatically generate a curriculum sequence. Experimental results on 3 sparse reward domains demonstrate that our task phasing approaches outperform state-of-the-art approaches with respect to performance.
  arXiv  Detail & Related papers  (2022-10-20T03:59:11Z)
• Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning [15.698612710580447]
  We propose a skill learning framework DR-GRL that aims to improve the sample efficiency and policy generalization. In a weakly supervised manner, we propose a Spatial Transform AutoEncoder (STAE) to learn an interpretable and controllable representation. We empirically demonstrate that DR-GRL significantly outperforms the previous methods in sample efficiency and policy generalization.
  arXiv  Detail & Related papers  (2022-02-28T09:05:14Z)
• Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency [61.03922379081648]
  We propose an off-policy sample efficient approach that requires no adversarial training or min-max optimization. Our empirical results show that D2-Imitation is effective in achieving good sample efficiency, outperforming several off-policy extension approaches of adversarial imitation.
  arXiv  Detail & Related papers  (2021-12-11T19:36:19Z)
• Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning [7.020079427649125]
  We show that grasping distinguishable skills for some tasks with non-unique optima can be essential for further improving its learning efficiency and performance. We propose a probabilistic mixture-of-experts (PMOE) for multimodal policy, together with a novel gradient estimator for the indifferentiability problem.
  arXiv  Detail & Related papers  (2021-04-19T08:21:56Z)
• DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation [57.358212277226315]
  In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. We propose a more data-efficient IfO algorithm
  arXiv  Detail & Related papers  (2021-03-31T23:46:32Z)
• Strictly Batch Imitation Learning by Energy-based Distribution Matching [104.33286163090179]
  Consider learning a policy purely on the basis of demonstrated behavior -- that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment. One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting. But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient. We argue that a good solution should be able to explicitly parameterize a policy, implicitly learn from rollout dynamics, and operate in an entirely offline fashion.
  arXiv  Detail & Related papers  (2020-06-25T03:27:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.