An Open-Source, Event-Driven Pipeline for Cryptocurrency Market Data: Ingestion, Forecasting, and On-Chain Fraud Detection

• URL: http://arxiv.org/abs/2608.29973v1
• Date: Sun, 30 Aug 2026 18:59:08 GMT
• Title: An Open-Source, Event-Driven Pipeline for Cryptocurrency Market Data: Ingestion, Forecasting, and On-Chain Fraud Detection
• Abstract summary: This paper describes a fully open-source pipeline that reproduces the behavior of a cloud-native, event-driven system.<n>We report the architecture, the modeling methodology, and the resulting metrics.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cryptocurrency markets generate high-frequency, multi-source data that is expensive to work with unless a team already has commercial-grade streaming and warehousing infrastructure in place. This paper describes a fully open-source pipeline that reproduces the behavior of a cloud-native, event-driven system -- file arrival triggering a message, a message triggering compute -- entirely on commodity hardware, using Apache Kafka and a filesystem-watching poller in place of managed cloud triggers. The pipeline partitions historical Gemini exchange data into hourly and minutely files, ingests them asynchronously through two independently grouped Kafka consumers (one for audit logging, one for Spark-triggered ETL), and lands cleaned output in a PostgreSQL warehouse with historical and aggregated schemas plus asset-specific data marts. We use the resulting Bitcoin data mart to compare a seasonal ARIMA model against a single-layer LSTM network for price forecasting, and separately apply Random Forest and Gradient Boosting classifiers, with additional engineered features, to the public Ethereum fraud detection benchmark introduced by Farrugia et al. We report the architecture, the modeling methodology, and the resulting metrics, and we are explicit about the limitations of comparing forecasts issued at different horizons and of evaluating fraud detection on a static, already-labeled dataset.

Related papers

• Sarus: Privacy-Preserving Multi-Vendor Perception Fusion via Homomorphic Encryption [0.20313422213594776]
  We present Sarus, a privacy-preserving framework for multi-vendor perception fusion via homomorphic encryption (HE)<n>Each vendor encodes detections as compact Gaussian moment vectors over a shared spatial lattice and transmits encrypted payloads to a fusion server.<n>The fused result is then decrypted and reconstructed into final detections through class-wise bin merging.
  arXiv  Detail & Related papers  (2026-07-21T14:38:52Z)
• DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams [48.434767119224375]
  We propose a paradigm shift towards Agentic Data Tailoring.<n>To overcome the data scarcity bottleneck in training such high-order capabilities, we design a two-stage pipeline.<n>$textDataClaw_0$-9B model synergizes Supervised Fine-Tuning with Group Relative Policy Optimization.
  arXiv  Detail & Related papers  (2026-06-19T11:31:43Z)
• ISOMORPH: A Supply Chain Digital Twin for Simulation, Dataset Generation, and Forecasting Benchmarks [6.803642669632622]
  We introduce ISOMORPH, the first public digital twin of a multi-echelon logistics network.<n>The simulator advances a directed routing graph in discrete time.<n>Three conservation laws encoded in the Markov chain serve as verification tools.
  arXiv  Detail & Related papers  (2026-05-12T21:31:32Z)
• GO-GenZip: Goal-Oriented Generative Sampling and Hybrid Compression [50.76941829813458]
  Current network data telemetry pipelines consist of massive streams of fine-grained Key Performance Indicators (KPIs) from multiple distributed sources towards central aggregators.<n>This work presents a generative AI (GenAI)-driven sampling and hybrid compression framework that redesigns network telemetry from a goal-oriented perspective.
  arXiv  Detail & Related papers  (2026-03-20T16:33:15Z)
• Causify DataFlow: A Framework For High-performance Machine Learning Stream Computing [0.0]
  We present DataFlow, a computational framework for building, testing, and deploying machine learning systems on unbounded time-series data.<n>Traditional data science assume finite datasets and require substantial reimplementation when moving from batch prototypes to streaming production systems.<n>DataFlow resolves these issues through a unified execution model based on acyclic graphs with point-in-time idempotency.
  arXiv  Detail & Related papers  (2025-12-30T04:24:04Z)
• Advanced fraud detection using machine learning models: enhancing financial transaction security [0.3370543514515051]
  This research presents an end-to-end, feature-rich machine learning framework for detecting credit card transaction anomalies and fraud using real-world data.
  arXiv  Detail & Related papers  (2025-06-12T15:59:25Z)
• CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking [85.68235482145091]
  Large-scale speech datasets have become valuable intellectual property.<n>We propose a novel dataset ownership verification method.<n>Our approach introduces a clustering-based backdoor watermark (CBW)<n>We conduct extensive experiments on benchmark datasets, verifying the effectiveness and robustness of our method against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-03-02T02:02:57Z)
• PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detection [51.20479454379662]
  We propose a. Federated Anomaly Detection framework named PeFAD with the increasing privacy concerns. We conduct extensive evaluations on four real datasets, where PeFAD outperforms existing state-of-the-art baselines by up to 28.74%.
  arXiv  Detail & Related papers  (2024-06-04T13:51:08Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• TensAIR: Real-Time Training of Neural Networks from Data-streams [1.409180142531996]
  This paper presents TensAIR, the first OL system for training ANNs in real time. TensAIR achieves remarkable performance and scalability by using a decentralized and asynchronous architecture to train ANN models. We empirically demonstrate that TensAIR achieves a nearly linear scale-out performance in terms of (1) the number of worker nodes deployed in the network, and (2) the throughput at which the data batches arrive.
  arXiv  Detail & Related papers  (2022-11-18T15:11:44Z)
• Scalable Teacher Forcing Network for Semi-Supervised Large Scale Data Streams [16.171151152904617]
  This paper offers Weakly Supervised Scalable Teacher Forcing Network (WeScatterNet) to cope with the scarcity of labelled samples and the large-scale data streams simultaneously. WeScatterNet is crafted under distributed computing platform of Apache Spark with a data-free model fusion strategy for model compression after parallel computing stage. The performance of WeScatterNet is numerically evaluated in the six large-scale data stream problems with only $25%$ label proportions.
  arXiv  Detail & Related papers  (2021-06-26T03:37:40Z)
• CREPO: An Open Repository to Benchmark Credal Network Algorithms [78.79752265884109]
  Credal networks are imprecise probabilistic graphical models based on, so-called credal, sets of probability mass functions. A Java library called CREMA has been recently released to model, process and query credal networks. We present CREPO, an open repository of synthetic credal networks, provided together with the exact results of inference tasks on these models.
  arXiv  Detail & Related papers  (2021-05-10T07:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.