End-to-End Large Portfolio Optimization for Variance Minimization with Neural Networks through Covariance Cleaning

• URL: http://arxiv.org/abs/2507.01918v2
• Date: Tue, 29 Jul 2025 04:20:02 GMT
• Title: End-to-End Large Portfolio Optimization for Variance Minimization with Neural Networks through Covariance Cleaning
• Authors: Christian Bongiorno, Efstratios Manolakis, Rosario Nunzio Mantegna,
• Abstract summary: We develop a rotation-invariant neural network that provides the global minimum-variance portfolio.<n>This explicit mathematical mapping offers clear interpretability of each module's role.<n>A single model can be calibrated on panels of a few hundred stocks and applied, without retraining, to one thousand US equities.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We develop a rotation-invariant neural network that provides the global minimum-variance portfolio by jointly learning how to lag-transform historical returns and how to regularise both the eigenvalues and the marginal volatilities of large equity covariance matrices. This explicit mathematical mapping offers clear interpretability of each module's role, so the model cannot be regarded as a pure black-box. The architecture mirrors the analytical form of the global minimum-variance solution yet remains agnostic to dimension, so a single model can be calibrated on panels of a few hundred stocks and applied, without retraining, to one thousand US equities-a cross-sectional jump that demonstrates robust out-of-sample generalisation. The loss function is the future realized minimum portfolio variance and is optimized end-to-end on real daily returns. In out-of-sample tests from January 2000 to December 2024 the estimator delivers systematically lower realised volatility, smaller maximum drawdowns, and higher Sharpe ratios than the best analytical competitors, including state-of-the-art non-linear shrinkage. Furthermore, although the model is trained end-to-end to produce an unconstrained (long-short) minimum-variance portfolio, we show that its learned covariance representation can be used in general optimizers under long-only constraints with virtually no loss in its performance advantage over competing estimators. These gains persist when the strategy is executed under a highly realistic implementation framework that models market orders at the auctions, empirical slippage, exchange fees, and financing charges for leverage, and they remain stable during episodes of acute market stress.

Related papers

• Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators [13.227055178509524]
  We propose a fault-tolerant evaluation framework that integrates bias and variance considerations within an adjustable tolerance level.<n>We show that proper calibration of $varepsilon$ ensures reliable evaluation across different variance regimes.<n> Experiments on real-world datasets demonstrate that our framework provides comprehensive and actionable insights into estimator behavior.
  arXiv  Detail & Related papers  (2026-02-06T22:14:46Z)
• A Novel approach to portfolio construction [0.0]
  This paper proposes a machine learning-based framework for asset selection and portfolio construction.<n>It is called the Best-Path Algorithm Sparse Graphical Model (BPASGM)<n>Monte Carlo simulations show BPASGM-based portfolios achieve more stable risk-return profiles, lower realized volatility, and superior risk-adjusted performance.
  arXiv  Detail & Related papers  (2026-02-03T09:52:06Z)
• Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum [62.691095807959215]
  We establish an optimal sample complexity of $O(-2)$ for obtaining an $$-optimal global policy using a single-timescale actor-critic (AC) algorithm.<n>These mechanisms are compatible with existing deep learning architectures and require only minor modifications, without compromising practical applicability.
  arXiv  Detail & Related papers  (2026-02-02T00:35:42Z)
• The Nonstationarity-Complexity Tradeoff in Return Prediction [5.8720142291102135]
  We investigate machine learning models for stock return prediction in non-stationary environments.<n>We show that a novel model selection method balances misspecification error, estimation variance, and non-stationarity, performing close to the best model in hindsight.
  arXiv  Detail & Related papers  (2025-12-29T16:49:19Z)
• Covariance-Aware Simplex Projection for Cardinality-Constrained Portfolio Optimization [0.0]
  Covariance-Aware Simplex Projection (CASP) is a two-stage repair operator that selects a target number of assets using volatility-normalized scores.<n>On S&P 500 data (2020-2024), CASP-Basic delivers materially lower portfolio variance than standard Euclidean repair.
  arXiv  Detail & Related papers  (2025-12-23T02:22:53Z)
• Integrated Prediction and Multi-period Portfolio Optimization [29.582959310549594]
  Multi-period portfolio optimization accounts for transaction costs, path-dependent risks, and the intertemporal structure of trading decisions.<n>This paper introduces IPMO, a model for multi-period mean-variance portfolio optimization with turnover penalties.<n>For scalability, we introduce a mirror-descent fixed-point (MDFP) differentiation scheme that avoids factorizing the Karush-Kuhn-Tucker (KKT) systems.
  arXiv  Detail & Related papers  (2025-12-12T04:31:22Z)
• ZIP-RC: Optimizing Test-Time Compute via Zero-Overhead Joint Reward-Cost Prediction [57.799425838564]
  We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost.<n> ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost.
  arXiv  Detail & Related papers  (2025-12-01T09:44:31Z)
• Modest-Align: Data-Efficient Alignment for Vision-Language Models [67.48633659305592]
  Cross-modal alignment models often suffer from overconfidence and degraded performance when operating in resource-constrained settings.<n>We propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency.<n>Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.
  arXiv  Detail & Related papers  (2025-10-24T16:11:10Z)
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs [53.77646961962239]
  Supervised Fine-Tuning (SFT) is a common approach to adapt Large Language Models (LLMs) to specialized tasks.<n>We show that SFT does not always hurt: using a smaller learning rate can substantially mitigate general performance degradation.
  arXiv  Detail & Related papers  (2025-09-25T05:28:22Z)
• Generalized Linear Bandits: Almost Optimal Regret with One-Pass Update [60.414548453838506]
  We study the generalized linear bandit (GLB) problem, a contextual multi-armed bandit framework that extends the classical linear model by incorporating a non-linear link function.<n>GLBs are widely applicable to real-world scenarios, but their non-linear nature introduces significant challenges in achieving both computational and statistical efficiency.<n>We propose a jointly efficient algorithm that attains a nearly optimal regret bound with $mathcalO(1)$ time and space complexities per round.
  arXiv  Detail & Related papers  (2025-07-16T02:24:21Z)
• Enhancing Black-Litterman Portfolio via Hybrid Forecasting Model Combining Multivariate Decomposition and Noise Reduction [13.04801847533423]
  We propose a novel hybrid forecasting model SSA-MAEMD-TCN to automate and improve the view generation process.<n> Empirical tests on the Nasdaq 100 Index stocks show a significant improvement in forecasting performance compared to baseline models.<n>The optimized portfolio performs well, with annualized returns and Sharpe ratios far exceeding those of the traditional portfolio.
  arXiv  Detail & Related papers  (2025-05-03T10:52:57Z)
• Feasible Learning [78.6167929413604]
  We introduce Feasible Learning (FL), a sample-centric learning paradigm where models are trained by solving a feasibility problem that bounds the loss for each training sample.<n>Our empirical analysis, spanning image classification, age regression, and preference optimization in large language models, demonstrates that models trained via FL can learn from data while displaying improved tail behavior compared to ERM, with only a marginal impact on average performance.
  arXiv  Detail & Related papers  (2025-01-24T20:39:38Z)
• Benign Overfitting in Out-of-Distribution Generalization of Linear Models [19.203753135860016]
  We take an initial step towards understanding benign overfitting in the Out-of-Distribution (OOD) regime.<n>We provide non-asymptotic guarantees proving that benign overfitting occurs in standard ridge regression.<n>We also present theoretical results for a more general family of target covariance matrix.
  arXiv  Detail & Related papers  (2024-12-19T02:47:39Z)
• Mean-Variance Portfolio Selection in Long-Term Investments with Unknown Distribution: Online Estimation, Risk Aversion under Ambiguity, and Universality of Algorithms [0.0]
  This paper adopts a perspective where data gradually and continuously reveal over time.<n>The original model is recast into an online learning framework, which is free from any statistical assumptions.<n>When the distribution of future data follows a normal shape, the growth rate of wealth is shown to increase by lifting the portfolio along the efficient frontier through the calibration of risk aversion.
  arXiv  Detail & Related papers  (2024-06-19T12:11:42Z)
• Portfolio Optimization with Robust Covariance and Conditional Value-at-Risk Constraints [0.0]
  We evaluated the performance of large-cap portfolio using various forms of Ledoit Shrinkage Covariance and Robust Gerber Covariance matrix. robustness estimators can outperform the market capitalization-weighted benchmark portfolio, particularly during bull markets. We incorporated unsupervised clustering algorithm K-means to the optimization algorithm.
  arXiv  Detail & Related papers  (2024-06-02T03:50:20Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction [54.21695754082441]
  Multi-step stock price prediction over a long-term horizon is crucial for forecasting its volatility. Current solutions to multi-step stock price prediction are mostly designed for single-step, classification-based predictions. We combine a deep hierarchical variational-autoencoder (VAE) and diffusion probabilistic techniques to do seq2seq stock prediction. Our model is shown to outperform state-of-the-art solutions in terms of its prediction accuracy and variance.
  arXiv  Detail & Related papers  (2023-08-18T16:21:15Z)
• Robustifying Markowitz [3.154269505086154]
  The heavy-tail characteristics of financial time series are in fact the cause for these erratic fluctuations of weights. We present a toolbox for stabilizing costs and weights for global minimum Markowitz portfolios. We demonstrate that robustified portfolios reach the lowest turnover compared to shrinkage-based and constrained portfolios.
  arXiv  Detail & Related papers  (2022-12-28T18:09:14Z)
• Improving Generalization via Uncertainty Driven Perturbations [107.45752065285821]
  We consider uncertainty-driven perturbations of the training data points. Unlike loss-driven perturbations, uncertainty-guided perturbations do not cross the decision boundary. We show that UDP is guaranteed to achieve the robustness margin decision on linear models.
  arXiv  Detail & Related papers  (2022-02-11T16:22:08Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.