Off-policy Evaluation for Payments at Adyen

• URL: http://arxiv.org/abs/2501.10470v1
• Date: Wed, 15 Jan 2025 22:17:01 GMT
• Title: Off-policy Evaluation for Payments at Adyen
• Authors: Alex Egg,
• Abstract summary: Off-Policy Evaluation (OPE) was applied to accelerate recommender system development and optimization at Adyen. Our analysis, conducted on a billion-scale dataset of transactions, reveals a strong correlation between OPE estimates and online A/B test results. We provide guidance on their effectiveness and integration into the decision-making systems for large-scale industrial payment systems.
• Score: 0.0
• License:
• Abstract: This paper demonstrates the successful application of Off-Policy Evaluation (OPE) to accelerate recommender system development and optimization at Adyen, a global leader in financial payment processing. Facing the limitations of traditional A/B testing, which proved slow, costly, and often inconclusive, we integrated OPE to enable rapid evaluation of new recommender system variants using historical data. Our analysis, conducted on a billion-scale dataset of transactions, reveals a strong correlation between OPE estimates and online A/B test results, projecting an incremental 9--54 million transactions over a six-month period. We explore the practical challenges and trade-offs associated with deploying OPE in a high-volume production environment, including leveraging exploration traffic for data collection, mitigating variance in importance sampling, and ensuring scalability through the use of Apache Spark. By benchmarking various OPE estimators, we provide guidance on their effectiveness and integration into the decision-making systems for large-scale industrial payment systems.

Related papers

• A Scalable Data-Driven Framework for Systematic Analysis of SEC 10-K Filings Using Large Language Models [0.0]
  We propose a novel data-driven approach to analyze and rate the performance of companies based on their SEC 10-K filings. The proposed scheme is then implemented on an interactive GUI as a no-code solution for running the data pipeline and creating the visualizations. The application showcases the rating results and provides year-on-year comparisons of company performance.
  arXiv  Detail & Related papers  (2024-09-26T06:57:22Z)
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline [78.00363373925758]
  We study the features of the BPR model, indicating their impact on its performance, and investigate open-source BPR implementations. Our analysis reveals inconsistencies between these implementations and the original BPR paper, leading to a significant decrease in performance of up to 50% for specific implementations. We show that the BPR model can achieve performance levels close to state-of-the-art methods on the top-n recommendation tasks and even outperform them on specific datasets.
  arXiv  Detail & Related papers  (2024-09-21T18:39:53Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)
• A Bargaining-based Approach for Feature Trading in Vertical Federated Learning [54.51890573369637]
  We propose a bargaining-based feature trading approach in Vertical Federated Learning (VFL) to encourage economically efficient transactions. Our model incorporates performance gain-based pricing, taking into account the revenue-based optimization objectives of both parties.
  arXiv  Detail & Related papers  (2024-02-23T10:21:07Z)
• Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation [17.319113169622806]
  Off-Policy Evaluation (OPE) aims to assess the effectiveness of counterfactual policies using only offline logged data. Existing evaluation metrics for OPE estimators primarily focus on the "accuracy" of OPE or that of downstream policy selection. We develop a new metric, called SharpeRatio@k, which measures the risk-return tradeoff of policy portfolios formed by an OPE estimator.
  arXiv  Detail & Related papers  (2023-11-30T02:56:49Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• Off-Policy Evaluation for Large Action Spaces via Embeddings [36.42838320396534]
  Off-policy evaluation (OPE) in contextual bandits has seen rapid adoption in real-world systems. Existing OPE estimators degrade severely when the number of actions is large. We propose a new OPE estimator that leverages marginalized importance weights when action embeddings provide structure in the action space.
  arXiv  Detail & Related papers  (2022-02-13T14:00:09Z)
• TTRS: Tinkoff Transactions Recommender System benchmark [62.997667081978825]
  We present the TTRS - Tinkoff Transactions Recommender System benchmark. This financial transaction benchmark contains over 2 million interactions between almost 10,000 users and more than 1,000 merchant brands over 14 months. We also present a comprehensive comparison of the current popular RecSys methods on the next-period recommendation task and conduct a detailed analysis of their performance against various metrics and recommendation goals.
  arXiv  Detail & Related papers  (2021-10-11T20:04:07Z)
• Evaluating the Robustness of Off-Policy Evaluation [10.760026478889664]
  Off-policy Evaluation (OPE) evaluates the performance of hypothetical policies leveraging only offline log data. It is particularly useful in applications where the online interaction involves high stakes and expensive setting. We develop Interpretable Evaluation for Offline Evaluation (IEOE), an experimental procedure to evaluate OPE estimators' robustness.
  arXiv  Detail & Related papers  (2021-08-31T09:33:13Z)
• Enhancing User' s Income Estimation with Super-App Alternative Data [59.60094442546867]
  It compares the performance of these alternative data sources with the performance of industry-accepted bureau income estimators. Ultimately, this paper shows the incentive for financial institutions to seek to incorporate alternative data into constructing their risk profiles.
  arXiv  Detail & Related papers  (2021-04-12T21:34:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.