Research on Optimizing Real-Time Data Processing in High-Frequency Trading Algorithms using Machine Learning

• URL: http://arxiv.org/abs/2412.01062v1
• Date: Mon, 02 Dec 2024 02:46:10 GMT
• Title: Research on Optimizing Real-Time Data Processing in High-Frequency Trading Algorithms using Machine Learning
• Authors: Yuxin Fan, Zhuohuan Hu, Lei Fu, Yu Cheng, Liyang Wang, Yuxiang Wang,
• Abstract summary: The objective of this work is to optimise the real-time processing of data in high-frequency trading algorithms.<n>The dynamic feature selection mechanism is responsible for monitoring and analysing market data in real time.<n>The lightweight neural networks are designed in a modular fashion, comprising fast convolutional layers and pruning techniques.
• Score: 19.43972851292453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-frequency trading (HFT) represents a pivotal and intensely competitive domain within the financial markets. The velocity and accuracy of data processing exert a direct influence on profitability, underscoring the significance of this field. The objective of this work is to optimise the real-time processing of data in high-frequency trading algorithms. The dynamic feature selection mechanism is responsible for monitoring and analysing market data in real time through clustering and feature weight analysis, with the objective of automatically selecting the most relevant features. This process employs an adaptive feature extraction method, which enables the system to respond and adjust its feature set in a timely manner when the data input changes, thus ensuring the efficient utilisation of data. The lightweight neural networks are designed in a modular fashion, comprising fast convolutional layers and pruning techniques that facilitate the expeditious completion of data processing and output prediction. In contrast to conventional deep learning models, the neural network architecture has been specifically designed to minimise the number of parameters and computational complexity, thereby markedly reducing the inference time. The experimental results demonstrate that the model is capable of maintaining consistent performance in the context of varying market conditions, thereby illustrating its advantages in terms of processing speed and revenue enhancement.

Related papers

• A Deep Learning Approach to Anomaly Detection in High-Frequency Trading Data [12.06584491356733]
  This paper proposes an algorithm based on a staged sliding window Transformer architecture to detect abnormal behaviors in the microstructure of the foreign exchange market. The method captures multi-scale temporal features through a staged sliding window, extracts global and local dependencies by combining the self-attention mechanism and weighted attention mechanism of the Transformer.
  arXiv  Detail & Related papers  (2025-03-31T23:14:31Z)
• Data-driven tool wear prediction in milling, based on a process-integrated single-sensor approach [1.6574413179773764]
  This study explores data-driven methods, in particular deep learning, for tool wear prediction. It investigates the transferability of predictive models using minimal training data, validated across two processes. The ConvNeXt model has an exceptional performance, achieving 99.1% accuracy in identifying tool wear.
  arXiv  Detail & Related papers  (2024-12-27T23:10:32Z)
• An Automated Data Mining Framework Using Autoencoders for Feature Extraction and Dimensionality Reduction [10.358417199718462]
  This study proposes an automated data mining framework based on autoencoders and experimentally verifies its effectiveness in feature extraction and data dimensionality reduction.<n>Through the encoding-decoding structure, the autoencoder can capture the data's potential characteristics and achieve noise reduction and anomaly detection.<n>In the future, with the advancement of deep learning and big data technology, the autoencoder method combined with a generative adversarial network (GAN) or graph neural network (GNN) is expected to be more widely used in the fields of complex data processing, real-time data analysis and intelligent decision-making.
  arXiv  Detail & Related papers  (2024-12-03T07:04:10Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• AdapFair: Ensuring Continuous Fairness for Machine Learning Operations [7.909259406397651]
  We present a debiasing framework designed to find an optimal fair transformation of input data. We leverage the normalizing flows to enable efficient, information-preserving data transformation. We introduce an efficient optimization algorithm with closed-formed gradient computations.
  arXiv  Detail & Related papers  (2024-09-23T15:01:47Z)
• Aeroengine performance prediction using a physical-embedded data-driven method [4.450340826544577]
  We propose a strategy that combines domain knowledge from both the aeroengine and neural network realms to enable real-time prediction of engine performance parameters. Leveraging aeroengine domain knowledge, we judiciously design the network structure and regulate the internal information flow. To rigorously evaluate the effectiveness and robustness of our proposed strategy, we conduct comprehensive validation across two distinct datasets.
  arXiv  Detail & Related papers  (2024-06-29T17:56:58Z)
• Automatic Input Feature Relevance via Spectral Neural Networks [0.9236074230806581]
  We propose a novel method to estimate the relative importance of the input components for a Deep Neural Network. This is achieved by leveraging on a spectral re-parametrization of the optimization process. The technique is successfully challenged against both synthetic and real data.
  arXiv  Detail & Related papers  (2024-06-03T10:39:12Z)
• Sparse Attention-driven Quality Prediction for Production Process Optimization in Digital Twins [53.70191138561039]
  We propose to deploy a digital twin of the production line by encoding its operational logic in a data-driven approach. We adopt a quality prediction model for production process based on self-attention-enabled temporal convolutional neural networks. Our operation experiments on a specific tobacco shredding line demonstrate that the proposed digital twin-based production process optimization method fosters seamless integration between virtual and real production lines.
  arXiv  Detail & Related papers  (2024-05-20T09:28:23Z)
• Switchable Decision: Dynamic Neural Generation Networks [98.61113699324429]
  We propose a switchable decision to accelerate inference by dynamically assigning resources for each data instance. Our method benefits from less cost during inference while keeping the same accuracy.
  arXiv  Detail & Related papers  (2024-05-07T17:44:54Z)
• Distributed Neural Representation for Reactive in situ Visualization [23.80657290203846]
  Implicit neural representations (INRs) have emerged as a powerful tool for compressing large-scale volume data. We develop a distributed neural representation and optimize it for in situ visualization. Our technique eliminates data exchanges between processes, achieving state-of-the-art compression speed, quality and ratios.
  arXiv  Detail & Related papers  (2023-03-28T03:55:47Z)
• Bilinear Input Normalization for Neural Networks in Financial Forecasting [101.89872650510074]
  We propose a novel data-driven normalization method for deep neural networks that handle high-frequency financial time-series. The proposed normalization scheme takes into account the bimodal characteristic of financial time-series. Our experiments, conducted with state-of-the-arts neural networks and high-frequency data, show significant improvements over other normalization techniques.
  arXiv  Detail & Related papers  (2021-09-01T07:52:03Z)
• Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data with Spatial Information [52.635997570873194]
  This work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to process complex multi-dimensional time series data with spatial information. The proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.
  arXiv  Detail & Related papers  (2021-01-12T20:08:18Z)
• Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity [57.275753974812666]
  Federated learning involves learning from data samples distributed across a network of clients while the data remains local. In this paper, we propose a novel straggler-resilient federated learning method that incorporates statistical characteristics of the clients' data to adaptively select the clients in order to speed up the learning procedure.
  arXiv  Detail & Related papers  (2020-12-28T19:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.