Counter-Empirical Attacking based on Adversarial Reinforcement Learning for Time-Relevant Scoring System

• URL: http://arxiv.org/abs/2311.05144v2
• Date: Tue, 19 Dec 2023 05:21:31 GMT
• Title: Counter-Empirical Attacking based on Adversarial Reinforcement Learning for Time-Relevant Scoring System
• Abstract summary: We propose a "counter-empirical attacking" mechanism that can generate "attacking" behavior traces and try to break the empirical rules of the scoring system. By training the adversarial learning problem, a proper scoring function can be learned to be robust to the attacking activity traces that are trying to violate the empirical criteria.
• Score: 22.86575572172689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scoring systems are commonly seen for platforms in the era of big data. From credit scoring systems in financial services to membership scores in E-commerce shopping platforms, platform managers use such systems to guide users towards the encouraged activity pattern, and manage resources more effectively and more efficiently thereby. To establish such scoring systems, several "empirical criteria" are firstly determined, followed by dedicated top-down design for each factor of the score, which usually requires enormous effort to adjust and tune the scoring function in the new application scenario. What's worse, many fresh projects usually have no ground-truth or any experience to evaluate a reasonable scoring system, making the designing even harder. To reduce the effort of manual adjustment of the scoring function in every new scoring system, we innovatively study the scoring system from the preset empirical criteria without any ground truth, and propose a novel framework to improve the system from scratch. In this paper, we propose a "counter-empirical attacking" mechanism that can generate "attacking" behavior traces and try to break the empirical rules of the scoring system. Then an adversarial "enhancer" is applied to evaluate the scoring system and find the improvement strategy. By training the adversarial learning problem, a proper scoring function can be learned to be robust to the attacking activity traces that are trying to violate the empirical criteria. Extensive experiments have been conducted on two scoring systems including a shared computing resource platform and a financial credit system. The experimental results have validated the effectiveness of our proposed framework.

Related papers

• Evaluation-driven Scaling for Scientific Discovery [77.20820317940581]
  We introduce Simple Test-time Evaluation-driven Scaling (SimpleTES), a framework that combines parallel exploration, feedback-driven refinement, and local selection.<n>We find substantial gains unlocked by scaling evaluation-driven discovery loops along the right dimensions.<n>Our results establish effective evaluation-driven loop scaling as a central axis for advancing scientific discovery.
  arXiv  Detail & Related papers  (2026-04-21T11:24:09Z)
• Score-Driven Rating System for Sports [0.0]
  This paper introduces a score-driven rating system, a generalization of the classical Elo rating system.<n>The proposed framework accommodates a wide range of game results, such as point differences, win/draw/loss outcomes, or complete rankings.
  arXiv  Detail & Related papers  (2026-04-10T09:22:33Z)
• Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors [0.0]
  This study investigates the effects of construct-irrelevant factors on a dual-architecture LLM-based scoring system.<n>It was found that the scoring system was generally robust to padding responses with meaningless text, spelling errors, and writing sophistication.
  arXiv  Detail & Related papers  (2026-03-26T17:29:20Z)
• The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning [54.67958855362658]
  We propose a dynamic framework that stress tests unlearning using complex structured queries.<n>Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains.<n>Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets.
  arXiv  Detail & Related papers  (2026-03-11T19:51:33Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• Collective dynamics of strategic classification [49.14942990735098]
  We apply evolutionary game theory to the problem of feedback loops between collectives of users and institutions.<n>We show that increased detection capabilities reduce social costs and could lead to users' improvement.<n>The speed at which the institutions re-adapt to the user's population plays a role in the final outcome.
  arXiv  Detail & Related papers  (2025-08-12T20:57:17Z)
• Re-evaluating Open-ended Evaluation of Large Language Models [50.23008729038318]
  We show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental. We propose evaluation as a 3-player game, and introduce novel game-theoretic solution concepts to ensure robustness to redundancy.
  arXiv  Detail & Related papers  (2025-02-27T15:07:47Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably. This poses a significant challenge to ensuring their safe deployment. We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• Learning Fair Ranking Policies via Differentiable Optimization of Ordered Weighted Averages [55.04219793298687]
  This paper shows how efficiently-solvable fair ranking models can be integrated into the training loop of Learning to Rank. In particular, this paper is the first to show how to backpropagate through constrained optimizations of OWA objectives, enabling their use in integrated prediction and decision models.
  arXiv  Detail & Related papers  (2024-02-07T20:53:53Z)
• Learning Recovery Strategies for Dynamic Self-healing in Reactive Systems [1.7218973692320518]
  Self-healing systems depend on following a set of predefined instructions to recover from a known failure state. Our proposal targets complex reactive systems, defining monitors as predicates specifying satisfiability conditions of system properties. We use a Reinforcement Learning-based technique to learn a recovery strategy based on users' corrective sequences.
  arXiv  Detail & Related papers  (2024-01-22T23:34:21Z)
• Universal Evasion Attacks on Summarization Scoring [0.3121997724420106]
  We place automatic scoring in the context of regression machine learning tasks. We perform evasion attacks to explore its robustness. A BERTScore backdoor is observed: a simple trigger can score higher than any automatic summarization method.
  arXiv  Detail & Related papers  (2022-10-25T18:25:47Z)
• Evaluating Machine Unlearning via Epistemic Uncertainty [78.27542864367821]
  This work presents an evaluation of Machine Unlearning algorithms based on uncertainty. This is the first definition of a general evaluation of our best knowledge.
  arXiv  Detail & Related papers  (2022-08-23T09:37:31Z)
• ReAct: Temporal Action Detection with Relational Queries [84.76646044604055]
  This work aims at advancing temporal action detection (TAD) using an encoder-decoder framework with action queries. We first propose a relational attention mechanism in the decoder, which guides the attention among queries based on their relations. Lastly, we propose to predict the localization quality of each action query at inference in order to distinguish high-quality queries.
  arXiv  Detail & Related papers  (2022-07-14T17:46:37Z)
• Large-Scale Sequential Learning for Recommender and Engineering Systems [91.3755431537592]
  In this thesis, we focus on the design of an automatic algorithms that provide personalized ranking by adapting to the current conditions. For the former, we propose novel algorithm called SAROS that take into account both kinds of feedback for learning over the sequence of interactions. The proposed idea of taking into account the neighbour lines shows statistically significant results in comparison with the initial approach for faults detection in power grid.
  arXiv  Detail & Related papers  (2022-05-13T21:09:41Z)
• Towards a Fairness-Aware Scoring System for Algorithmic Decision-Making [35.21763166288736]
  We propose a general framework to create data-driven fairness-aware scoring systems. We show that the proposed framework provides practitioners or policymakers great flexibility to select their desired fairness requirements.
  arXiv  Detail & Related papers  (2021-09-21T09:46:35Z)
• Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples [29.385242714424624]
  evaluating robustness of machine-learning models to adversarial examples is a challenging problem. We define a set of quantitative indicators which unveil common failures in the optimization of gradient-based attacks. Our experimental analysis shows that the proposed indicators of failure can be used to visualize, debug and improve current adversarial robustness evaluations.
  arXiv  Detail & Related papers  (2021-06-18T06:57:58Z)
• Robustness Gym: Unifying the NLP Evaluation Landscape [91.80175115162218]
  Deep neural networks are often brittle when deployed in real-world systems. Recent research has focused on testing the robustness of such models. We propose a solution in the form of Robustness Gym, a simple and evaluation toolkit.
  arXiv  Detail & Related papers  (2021-01-13T02:37:54Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• Active Bayesian Assessment for Black-Box Classifiers [20.668691047355072]
  We introduce an active Bayesian approach for assessment of classifier performance to satisfy the desiderata of both reliability and label-efficiency. We first develop inference strategies to quantify uncertainty for common assessment metrics such as accuracy, misclassification cost, and calibration error. We then propose a general framework for active Bayesian assessment using inferred uncertainty to guide efficient selection of instances for labeling.
  arXiv  Detail & Related papers  (2020-02-16T08:08:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.