Factor-MCLS: Multi-agent learning system with reward factor matrix and multi-critic framework for dynamic portfolio optimization

• URL: http://arxiv.org/abs/2504.11874v1
• Date: Wed, 16 Apr 2025 08:51:09 GMT
• Title: Factor-MCLS: Multi-agent learning system with reward factor matrix and multi-critic framework for dynamic portfolio optimization
• Authors: Ruoyu Sun, Angelos Stefanidis, Zhengyong Jiang, Jionglong Su,
• Abstract summary: It is difficult for investors to intervene in the training based on different levels of risk aversion towards each portfolio asset.<n>Existing deep reinforcement learning (DRL) agents may not develop a thorough understanding of the factors responsible for the portfolio return and risk.<n>We propose a reward factor matrix for elucidating each asset in each portfolio.<n>In this way, our DRL-based learning system can effectively learn the factors influencing portfolio return and risk.
• Score: 4.36011451106729
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Typical deep reinforcement learning (DRL) agents for dynamic portfolio optimization learn the factors influencing portfolio return and risk by analyzing the output values of the reward function while adjusting portfolio weights within the training environment. However, it faces a major limitation where it is difficult for investors to intervene in the training based on different levels of risk aversion towards each portfolio asset. This difficulty arises from another limitation: existing DRL agents may not develop a thorough understanding of the factors responsible for the portfolio return and risk by only learning from the output of the reward function. As a result, the strategy for determining the target portfolio weights is entirely dependent on the DRL agents themselves. To address these limitations, we propose a reward factor matrix for elucidating the return and risk of each asset in the portfolio. Additionally, we propose a novel learning system named Factor-MCLS using a multi-critic framework that facilitates learning of the reward factor matrix. In this way, our DRL-based learning system can effectively learn the factors influencing portfolio return and risk. Moreover, based on the critic networks within the multi-critic framework, we develop a risk constraint term in the training objective function of the policy function. This risk constraint term allows investors to intervene in the training of the DRL agent according to their individual levels of risk aversion towards the portfolio assets.

Related papers

• STORM: A Spatio-Temporal Factor Model Based on Dual Vector Quantized Variational Autoencoders for Financial Trading [55.02735046724146]
  In financial trading, factor models are widely used to price assets and capture excess returns from mispricing. We propose a Spatio-Temporal factOR Model based on dual vector quantized variational autoencoders, named STORM. Storm extracts features of stocks from temporal and spatial perspectives, then fuses and aligns these features at the fine-grained and semantic level, and represents the factors as multi-dimensional embeddings.
  arXiv  Detail & Related papers  (2024-12-12T17:15:49Z)
• MILLION: A General Multi-Objective Framework with Controllable Risk for Portfolio Management [16.797109778036862]
  We propose a general Multi-objectIve framework with controLLableIsk for pOrtfolio maMILLION.<n>In the risk control phase, we propose two methods, i.e., portfolio adaption and portfolio improvement.<n>The results demonstrate the effectiveness and efficiency of the proposed framework.
  arXiv  Detail & Related papers  (2024-12-04T05:19:34Z)
• VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment [66.80143024475635]
  We propose VinePPO, a straightforward approach to compute unbiased Monte Carlo-based estimates. We show that VinePPO consistently outperforms PPO and other RL-free baselines across MATH and GSM8K datasets.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• Deep Reinforcement Learning and Mean-Variance Strategies for Responsible Portfolio Optimization [49.396692286192206]
  We study the use of deep reinforcement learning for responsible portfolio optimization by incorporating ESG states and objectives. Our results show that deep reinforcement learning policies can provide competitive performance against mean-variance approaches for responsible portfolio allocation.
  arXiv  Detail & Related papers  (2024-03-25T12:04:03Z)
• Combining Transformer based Deep Reinforcement Learning with Black-Litterman Model for Portfolio Optimization [0.0]
  As a model-free algorithm, deep reinforcement learning (DRL) agent learns and makes decisions by interacting with the environment in an unsupervised way. We propose a hybrid portfolio optimization model combining the DRL agent and the Black-Litterman (BL) model. Our DRL agent significantly outperforms various comparison portfolio choice strategies and alternative DRL frameworks by at least 42% in terms of accumulated return.
  arXiv  Detail & Related papers  (2024-02-23T16:01:37Z)
• RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization [49.26510528455664]
  We introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. We show that RiskQ can obtain promising performance through extensive experiments.
  arXiv  Detail & Related papers  (2023-11-03T07:18:36Z)
• Optimizing Credit Limit Adjustments Under Adversarial Goals Using Reinforcement Learning [42.303733194571905]
  We seek to find and automatize an optimal credit card limit adjustment policy by employing reinforcement learning techniques. Our research establishes a conceptual structure for applying reinforcement learning framework to credit limit adjustment.
  arXiv  Detail & Related papers  (2023-06-27T16:10:36Z)
• Factor Investing with a Deep Multi-Factor Model [123.52358449455231]
  We develop a novel deep multi-factor model that adopts industry neutralization and market neutralization modules with clear financial insights. Tests on real-world stock market data demonstrate the effectiveness of our deep multi-factor model.
  arXiv  Detail & Related papers  (2022-10-22T14:47:11Z)
• Efficient Risk-Averse Reinforcement Learning [79.61412643761034]
  In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns. We prove that under certain conditions this inevitably leads to a local-optimum barrier, and propose a soft risk mechanism to bypass it. We demonstrate improved risk aversion in maze navigation, autonomous driving, and resource allocation benchmarks.
  arXiv  Detail & Related papers  (2022-05-10T19:40:52Z)
• Risk budget portfolios with convex Non-negative Matrix Factorization [0.0]
  We propose a portfolio allocation method based on risk factor budgeting using convex Nonnegative Matrix Factorization (NMF) We evaluate our method in the context of volatility targeting on two long-only global portfolios of cryptocurrencies and traditional assets.
  arXiv  Detail & Related papers  (2022-04-06T12:02:15Z)
• Deep Risk Model: A Deep Learning Solution for Mining Latent Risk Factors to Improve Covariance Matrix Estimation [8.617532047238461]
  We propose a deep learning solution to effectively "design" risk factors with neural networks. Our method can obtain $1.9%$ higher explained variance measured by $R2$ and also reduce the risk of a global minimum variance portfolio.
  arXiv  Detail & Related papers  (2021-07-12T05:30:50Z)
• Learning Risk Preferences from Investment Portfolios Using Inverse Optimization [25.19470942583387]
  This paper presents a novel approach of measuring risk preference from existing portfolios using inverse optimization. We demonstrate our methods on real market data that consists of 20 years of asset pricing and 10 years of mutual fund portfolio holdings.
  arXiv  Detail & Related papers  (2020-10-04T21:29:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.