Beyond Accuracy: Measuring Logical Compliance of Predictive Models

• URL: http://arxiv.org/abs/2606.20208v1
• Date: Thu, 18 Jun 2026 13:21:04 GMT
• Title: Beyond Accuracy: Measuring Logical Compliance of Predictive Models
• Abstract summary: Rule Violation Score (RVS) is a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules.<n>We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression.<n>Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance.
• Score: 2.963223599781967
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning models are predominantly evaluated through predictive performance metrics such as ranking quality, prediction error, or classification accuracy. While these metrics effectively quantify how closely predictions match the ground truth, they do not assess whether model outputs respect predefined logical or domain-specific constraints. In high-stakes applications, including healthcare, finance, and autonomous systems, logical consistency can be as critical as predictive accuracy, yet no standard metric captures this dimension. We introduce the Rule Violation Score (RVS), a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules, independently of predictive accuracy. RVS treats hard rules (strict constraints) and soft rules (statistical regularities) differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules. Beyond evaluating models, RVS can also evaluate the logical consistency of training datasets and help identify poorly defined rules. We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance, revealing differences in model behavior that standard metrics fail to capture.

Related papers

• Conformal Reliability: A New Evaluation Metric for Conditional Generation [62.761166941396844]
  Conditional generative models have recently remarkable success in various applications.<n>We propose a novel evaluation metric called reliability based on prediction set at a pre-specified confidence level.<n>We introduce Conformal ReLiability (CReL), a framework that can (i) construct a prediction set and (ii) accurately optimize the reliability score within the constructed prediction set.
  arXiv  Detail & Related papers  (2026-05-29T03:52:44Z)
• First-Order Efficiency for Probabilistic Value Estimation via A Statistical Viewpoint [18.696678371045216]
  We propose an efficiency-Aware Surrogate-adjusted Estimator (EASE) that directly chooses the sampling law and surrogate to minimize the first-order MSE.<n>We demonstrate that EASE consistently outperforms state-of-the-art estimators for various probabilistic values.
  arXiv  Detail & Related papers  (2026-05-04T17:02:17Z)
• A Theoretical Framework for Statistical Evaluability of Generative Models [57.9316356505791]
  We introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics.<n>We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs) and Rényi divergences.<n>We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors.<n>In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events.
  arXiv  Detail & Related papers  (2026-04-07T01:53:59Z)
• Empirical Characterization of Rationale Stability Under Controlled Perturbations for Explainable Pattern Recognition [0.876171257490837]
  We propose a novel metric aimed at assessing the consistency of model explanations.<n>We implement this metric using a pre-trained BERT model on the SST-2 sentiment analysis dataset.<n>We evaluate the ability of this metric to identify misaligned predictions and inconsistencies in model explanations.
  arXiv  Detail & Related papers  (2026-04-06T06:05:21Z)
• What Do Learned Models Measure? [0.0]
  In many scientific and data-driven applications, machine learning models are increasingly used as measurement instruments.<n>We show that standard evaluation criteria in machine learning, including generalization error, calibration, and robustness, do not guarantee measurement stability.
  arXiv  Detail & Related papers  (2026-01-26T09:00:48Z)
• Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking [46.95596181965493]
  This paper presents a unified and practical framework for unsupervised model evaluation and ranking.<n>We show that hybrid metrics consistently outperform single-aspect metrics on both dataset-centric and model-centric evaluation settings.
  arXiv  Detail & Related papers  (2025-10-03T12:48:11Z)
• Uncertainty Quantification for Regression using Proper Scoring Rules [76.24649098854219]
  We introduce a unified UQ framework for regression based on proper scoring rules, such as CRPS, logarithmic, squared error, and quadratic scores.<n>We derive closed-form expressions for the uncertainty measures under practical parametric assumptions and show how to estimate them using ensembles of models.<n>Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable UQ measures.
  arXiv  Detail & Related papers  (2025-09-30T17:52:12Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Rigorous Assessment of Model Inference Accuracy using Language Cardinality [5.584832154027001]
  We develop a systematic approach that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools.
  arXiv  Detail & Related papers  (2022-11-29T21:03:26Z)
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning [123.3472310767721]
  Prediction credibility measures are fundamental in statistics and machine learning. These measures should account for the wide variety of models used in practice. The framework developed in this work expresses the credibility as a risk-fit trade-off.
  arXiv  Detail & Related papers  (2020-11-24T19:52:38Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.