Reinforcement Learning with Hidden Markov Models for Discovering
Decision-Making Dynamics
- URL: http://arxiv.org/abs/2401.13929v1
- Date: Thu, 25 Jan 2024 04:03:32 GMT
- Title: Reinforcement Learning with Hidden Markov Models for Discovering
Decision-Making Dynamics
- Authors: Xingche Guo, Donglin Zeng, Yuanjia Wang
- Abstract summary: Evidence indicates that reward processing abnormalities may serve as a behavioral marker for MDD.
Recent findings suggest the inadequacy of characterizing reward learning solely based on a single RL model.
We propose a novel RL-HMM framework for analyzing reward-based decision-making.
- Score: 6.582785642715135
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Major depressive disorder (MDD) presents challenges in diagnosis and
treatment due to its complex and heterogeneous nature. Emerging evidence
indicates that reward processing abnormalities may serve as a behavioral marker
for MDD. To measure reward processing, patients perform computer-based
behavioral tasks that involve making choices or responding to stimulants that
are associated with different outcomes. Reinforcement learning (RL) models are
fitted to extract parameters that measure various aspects of reward processing
to characterize how patients make decisions in behavioral tasks. Recent
findings suggest the inadequacy of characterizing reward learning solely based
on a single RL model; instead, there may be a switching of decision-making
processes between multiple strategies. An important scientific question is how
the dynamics of learning strategies in decision-making affect the reward
learning ability of individuals with MDD. Motivated by the probabilistic reward
task (PRT) within the EMBARC study, we propose a novel RL-HMM framework for
analyzing reward-based decision-making. Our model accommodates learning
strategy switching between two distinct approaches under a hidden Markov model
(HMM): subjects making decisions based on the RL model or opting for random
choices. We account for continuous RL state space and allow time-varying
transition probabilities in the HMM. We introduce a computationally efficient
EM algorithm for parameter estimation and employ a nonparametric bootstrap for
inference. We apply our approach to the EMBARC study to show that MDD patients
are less engaged in RL compared to the healthy controls, and engagement is
associated with brain activities in the negative affect circuitry during an
emotional conflict task.
Related papers
- Querying Easily Flip-flopped Samples for Deep Active Learning [63.62397322172216]
Active learning is a machine learning paradigm that aims to improve the performance of a model by strategically selecting and querying unlabeled data.
One effective selection strategy is to base it on the model's predictive uncertainty, which can be interpreted as a measure of how informative a sample is.
This paper proposes the it least disagree metric (LDM) as the smallest probability of disagreement of the predicted label.
arXiv Detail & Related papers (2024-01-18T08:12:23Z) - Cross-modality Guidance-aided Multi-modal Learning with Dual Attention
for MRI Brain Tumor Grading [47.50733518140625]
Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly.
We propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading.
arXiv Detail & Related papers (2024-01-17T07:54:49Z) - Latent Variable Representation for Reinforcement Learning [131.03944557979725]
It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning.
We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle.
In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
arXiv Detail & Related papers (2022-12-17T00:26:31Z) - GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP,
and Beyond [101.5329678997916]
We study sample efficient reinforcement learning (RL) under the general framework of interactive decision making.
We propose a novel complexity measure, generalized eluder coefficient (GEC), which characterizes the fundamental tradeoff between exploration and exploitation.
We show that RL problems with low GEC form a remarkably rich class, which subsumes low Bellman eluder dimension problems, bilinear class, low witness rank problems, PO-bilinear class, and generalized regular PSR.
arXiv Detail & Related papers (2022-11-03T16:42:40Z) - Offline Reinforcement Learning with Instrumental Variables in Confounded
Markov Decision Processes [93.61202366677526]
We study the offline reinforcement learning (RL) in the face of unmeasured confounders.
We propose various policy learning methods with the finite-sample suboptimality guarantee of finding the optimal in-class policy.
arXiv Detail & Related papers (2022-09-18T22:03:55Z) - MRI-based Multi-task Decoupling Learning for Alzheimer's Disease
Detection and MMSE Score Prediction: A Multi-site Validation [9.427540028148963]
Accurately detecting Alzheimer's disease (AD) and predicting mini-mental state examination (MMSE) score are important tasks in elderly health by magnetic resonance imaging (MRI)
Most of the previous methods on these two tasks are based on single-task learning and rarely consider the correlation between them.
We propose a MRI-based multi-task decoupled learning method for AD detection and MMSE score prediction.
arXiv Detail & Related papers (2022-04-02T09:19:18Z) - Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approach [0.0]
Dynamic Treatment Regimes (DTRs) are widely studied to formalize this process.
We develop Reinforcement Learning methods to efficiently learn optimal treatment regimes.
arXiv Detail & Related papers (2021-12-08T20:22:04Z) - Identification of brain states, transitions, and communities using
functional MRI [0.5872014229110214]
We propose a Bayesian model-based characterization of latent brain states and showcase a novel method based on posterior predictive discrepancy.
Our results obtained through an analysis of task-fMRI data show appropriate lags between external task demands and change-points between brain states.
arXiv Detail & Related papers (2021-01-26T08:10:00Z) - Adversarial Sample Enhanced Domain Adaptation: A Case Study on
Predictive Modeling with Electronic Health Records [57.75125067744978]
We propose a data augmentation method to facilitate domain adaptation.
adversarially generated samples are used during domain adaptation.
Results confirm the effectiveness of our method and the generality on different tasks.
arXiv Detail & Related papers (2021-01-13T03:20:20Z) - On the Reliability and Generalizability of Brain-inspired Reinforcement
Learning Algorithms [10.09712608508383]
We show that the computational model combining model-based and model-free control, which we term the prefrontal RL, reliably encodes the information of high-level policy that humans learned.
This is the first attempt to formally test the possibility that computational models mimicking the way the brain solves general problems can lead to practical solutions.
arXiv Detail & Related papers (2020-07-09T06:32:42Z) - Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits
and RL [28.38826379640553]
We propose a more general and flexible parametric framework for sequential decision making.
Inspired by the known reward processing abnormalities of many mental disorders, our clinically-inspired agents demonstrated interesting behavioral trajectories.
arXiv Detail & Related papers (2020-05-10T01:43:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.