Improving S&P 500 Volatility Forecasting through Regime-Switching Methods

• URL: http://arxiv.org/abs/2510.03236v1
• Date: Sun, 21 Sep 2025 19:40:47 GMT
• Title: Improving S&P 500 Volatility Forecasting through Regime-Switching Methods
• Authors: Ava C. Blake, Nivika A. Gandhi, Anurag R. Jakkula,
• Abstract summary: We propose a multitude of regime-switching methods to improve the prediction of S&P 500 volatility.<n>To enhance forecast accuracy, we engineered features to capture both historical dynamics and forward-looking market sentiment.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate prediction of financial market volatility is critical for risk management, derivatives pricing, and investment strategy. In this study, we propose a multitude of regime-switching methods to improve the prediction of S&P 500 volatility by capturing structural changes in the market across time. We use eleven years of SPX data, from May 1st, 2014 to May 27th, 2025, to compute daily realized volatility (RV) from 5-minute intraday log returns, adjusted for irregular trading days. To enhance forecast accuracy, we engineered features to capture both historical dynamics and forward-looking market sentiment across regimes. The regime-switching methods include a soft Markov switching algorithm to estimate soft-regime probabilities, a distributional spectral clustering method that uses XGBoost to assign clusters at prediction time, and a coefficient-based soft regime algorithm that extracts HAR coefficients from time segments segmented through the Mood test and clusters through Bayesian GMM for soft regime weights, using XGBoost to predict regime probabilities. Models were evaluated across three time periods--before, during, and after the COVID-19 pandemic. The coefficient-based clustering algorithm outperformed all other models, including the baseline autoregressive model, during all time periods. Additionally, each model was evaluated on its recursive forecasting performance for 5- and 10-day horizons during each time period. The findings of this study demonstrate the value of regime-aware modeling frameworks and soft clustering approaches in improving volatility forecasting, especially during periods of heightened uncertainty and structural change.

Related papers

• Stochastic Volatility Modelling with LSTM Networks: A Hybrid Approach for S&P 500 Index Volatility Forecasting [0.0]
  This study proposes a hybrid modelling framework that integrates a volatility model with a Long Short Term Memory neural network.<n>The SV model improves statistical precision and captures latent volatility dynamics, especially in response to unforeseen events.<n>The LSTM network enhances the model's ability to detect complex nonlinear patterns in financial time series.
  arXiv  Detail & Related papers  (2025-12-13T09:21:43Z)
• Reinforcement Learning from Probabilistic Forecasts for Safe Decision-Making via Conditional Value-at-Risk Planning [41.52380204321823]
  This paper presents the Uncertainty-Aware Markov Decision Process (UAMDP), a unified framework that couples Bayesian forecasting, posterior-sampling reinforcement learning, and planning.<n>We evaluate UAMDP in two domains-high-frequency equity trading and retail inventory control-both marked by structural uncertainty and economic volatility.
  arXiv  Detail & Related papers  (2025-10-09T13:46:32Z)
• Geometric-Mean Policy Optimization [117.05113769757172]
  Group Relative Policy Optimization ( GRPO) has significantly enhanced the reasoning capability of large language models.<n> GRPO is observed to suffer from unstable policy updates when facing tokens with outlier importance-weighted rewards.<n>We propose Geometric-Mean Policy Optimization (GMPO) to improve the stability of GRPO through suppressing token reward outliers.
  arXiv  Detail & Related papers  (2025-07-28T09:54:05Z)
• Enhanced forecasting of stock prices based on variational mode decomposition, PatchTST, and adaptive scale-weighted layer [1.9635048365486127]
  This study introduces a novel composite forecasting framework that integrates variational mode decomposition (VMD), PatchTST, and adaptive scale-weighted layer (ASWL) The VMD-PatchTST-ASWL framework demonstrates significant improvements in forecasting accuracy compared to traditional models. This innovative approach provides a powerful tool for stock index price forecasting, with potential applications in various financial analysis and investment decision-making contexts.
  arXiv  Detail & Related papers  (2024-08-29T17:00:47Z)
• The Hybrid Forecast of S&P 500 Volatility ensembled from VIX, GARCH and LSTM models [0.0]
  This study explores four methods to improve the accuracy of volatility forecasts for the S&P 500. We find that machine learning approaches, particularly the hybrid LSTM models, significantly outperform the traditional GARCH model. The results of this study offer valuable insights for achieving more accurate volatility predictions.
  arXiv  Detail & Related papers  (2024-07-23T18:28:16Z)
• Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction [54.21695754082441]
  Multi-step stock price prediction over a long-term horizon is crucial for forecasting its volatility. Current solutions to multi-step stock price prediction are mostly designed for single-step, classification-based predictions. We combine a deep hierarchical variational-autoencoder (VAE) and diffusion probabilistic techniques to do seq2seq stock prediction. Our model is shown to outperform state-of-the-art solutions in terms of its prediction accuracy and variance.
  arXiv  Detail & Related papers  (2023-08-18T16:21:15Z)
• FuXi: A cascade machine learning forecasting system for 15-day global weather forecast [34.812266901884996]
  FuXi is a cascaded ML weather forecasting system that provides 15-day global forecasts with a temporal resolution of 6 hours and a spatial resolution of 0.25 degree. FuXi has comparable forecast performance to ECMWF EM in 15-day forecasts, making FuXi the first ML-based weather forecasting system to accomplish this achievement.
  arXiv  Detail & Related papers  (2023-06-22T13:34:26Z)
• Deep Learning Enhanced Realized GARCH [6.211385208178938]
  We propose a new approach to volatility modeling by combining deep learning (LSTM) and realized volatility measures. This LSTM-enhanced realized GARCH framework incorporates and distills modeling advances from financial econometrics, high frequency trading data and deep learning.
  arXiv  Detail & Related papers  (2023-02-16T00:20:43Z)
• DeepVol: Volatility Forecasting from High-Frequency Data with Dilated Causal Convolutions [53.37679435230207]
  We propose DeepVol, a model based on Dilated Causal Convolutions that uses high-frequency data to forecast day-ahead volatility. Our empirical results suggest that the proposed deep learning-based approach effectively learns global features from high-frequency data.
  arXiv  Detail & Related papers  (2022-09-23T16:13:47Z)
• Regime-based Implied Stochastic Volatility Model for Crypto Option Pricing [0.0]
  Existing methodologies fail to cope with the volatile nature of the emerging Digital Assets (DAs) We leverage recent advances in market regime (MR) clustering with the Implied volatility Model (ISVM) ISVM can incorporate investor expectations in each of the sentiment-driven periods by using implied volatility (IV) data. We demonstrate that MR-ISVM contributes to overcome the burden of complex adaption to jumps in higher order characteristics of option pricing models.
  arXiv  Detail & Related papers  (2022-08-15T15:31:42Z)
• Volatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes [36.712632126776285]
  We show how to re-cast a class of volatility models as a hierarchical Gaussian process (GP) model with specialized covariance functions. Within this framework, we take inspiration from well studied domains to introduce a new class of models, Volt and Magpie, that significantly outperform baselines in stock and wind speed forecasting.
  arXiv  Detail & Related papers  (2022-07-13T23:02:54Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.