PRAGMA: Revolut Foundation Model

• URL: http://arxiv.org/abs/2604.08649v1
• Date: Thu, 09 Apr 2026 18:00:00 GMT
• Title: PRAGMA: Revolut Foundation Model
• Abstract summary: PRAGMA is a family of foundation models for multi-source banking event sequences.<n>It supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction.
• Score: 3.159538292278343
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for multi-source banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.

Related papers

• A Foundation Model for Multimodal Event Sequences in Financial Applications [34.57471442552313]
  We present an approach based on pretraining a foundation transformer model on multimodal sequences of user events.<n>The proposed system outperforms traditional task-specific models while reducing development overhead.<n>The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.
  arXiv  Detail & Related papers  (2026-07-10T20:19:17Z)
• TokaMind: A Multi-Modal Transformer Foundation Model for Tokamak Plasma Dynamics [56.073642366268764]
  TokaMind is an open-source foundation model framework for fusion plasma modeling.<n>It is trained on heterogeneous tokamak diagnostics from the publicly available MAST dataset.<n>We evaluate TokaMind on the recently introduced MAST benchmark TokaMark.
  arXiv  Detail & Related papers  (2026-02-16T12:26:07Z)
• Dynamic Generation of Multi-LLM Agents Communication Topologies with Graph Diffusion Models [99.85131798240808]
  We introduce a novel generative framework called textitGuided Topology Diffusion (GTD)<n>Inspired by conditional discrete graph diffusion models, GTD formulates topology synthesis as an iterative construction process.<n>At each step, the generation is steered by a lightweight proxy model that predicts multi-objective rewards.<n>Experiments show that GTD can generate highly task-adaptive, sparse, and efficient communication topologies.
  arXiv  Detail & Related papers  (2025-10-09T05:28:28Z)
• MASS: Muli-agent simulation scaling for portfolio construction [17.363056358369143]
  We introduce the Multi-Agent Scaling Simulation (MASS), a novel framework that leverages multi-agent simulation for direct, end-to-end portfolio construction.<n>At its core, MASS employs a backward optimization process to dynamically learn the optimal distribution of heterogeneous agents.<n>We demonstrate that as the number of agents increases exponentially (up to 512), the aggregated decisions yield progressively higher excess returns.
  arXiv  Detail & Related papers  (2025-05-15T13:27:18Z)
• LENS: Large Pre-trained Transformer for Exploring Financial Time Series Regularities [3.475989206546412]
  We propose textbfLENS, a pre-trained model for financial time series.<n>textbfLENS effectively captures the complexity of financial systems through a carefully crafted model architecture.<n>Our work offers practical insights into developing pre-trained time series models in high-noise environments.
  arXiv  Detail & Related papers  (2024-08-19T15:59:46Z)
• Advancing Anomaly Detection: Non-Semantic Financial Data Encoding with LLMs [49.57641083688934]
  We introduce a novel approach to anomaly detection in financial data using Large Language Models (LLMs) embeddings. Our experiments demonstrate that LLMs contribute valuable information to anomaly detection as our models outperform the baselines.
  arXiv  Detail & Related papers  (2024-06-05T20:19:09Z)
• Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences [0.0]
  We present a generative pretraining method that can be used to obtain contextualised embeddings of financial transactions. We additionally perform large-scale pretraining of an embedding model using a corpus of data from 180 issuing banks containing 5.1 billion transactions.
  arXiv  Detail & Related papers  (2024-01-03T09:32:48Z)
• Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network [7.54290390842336]
  We propose an end-to-end autoregressive generative model that generates tokenized limit order book (LOB) messages. Using NASDAQ equity LOBs, we develop a custom tokenizer for message data, converting groups of successive digits to tokens. Results show promising performance in approximating the data distribution, as evidenced by low model perplexity.
  arXiv  Detail & Related papers  (2023-08-23T09:37:22Z)
• Slimmable Domain Adaptation [112.19652651687402]
  We introduce a simple framework, Slimmable Domain Adaptation, to improve cross-domain generalization with a weight-sharing model bank. Our framework surpasses other competing approaches by a very large margin on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T06:28:04Z)
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference [55.35176938713946]
  We develop deep autoencoding topic model (DATM) that uses a hierarchy of gamma distributions to construct its multi-stochastic-layer generative network. We propose a Weibull upward-downward variational encoder that deterministically propagates information upward via a deep neural network, followed by a downward generative model. The efficacy and scalability of our models are demonstrated on both unsupervised and supervised learning tasks on big corpora.
  arXiv  Detail & Related papers  (2020-06-15T22:22:56Z)
• CoLES: Contrastive Learning for Event Sequences with Self-Supervision [63.3568071938238]
  We address the problem of self-supervised learning on discrete event sequences generated by real-world users. We propose a new method "CoLES", which adapts contrastive learning, previously used for audio and computer vision domains.
  arXiv  Detail & Related papers  (2020-02-19T15:15:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.