Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation

• URL: http://arxiv.org/abs/2603.23517v1
• Date: Fri, 06 Mar 2026 03:50:59 GMT
• Title: Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation
• Abstract summary: We argue for a mechanism-aware evaluation that combines task-relevant symbolic rules with interpretability.<n>Standard evaluation shows the memorization achieves 94% field-name accuracy on unseen data, falsely suggesting competence.<n>Our symbolic-stic evaluation reveals this model violates core schema generalization rules, a failure invisible to accuracy metrics.
• Score: 8.169425244574203
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accuracy-based evaluation cannot reliably distinguish genuine generalization from shortcuts like memorization, leakage, or brittle heuristics, especially in small-data regimes. In this position paper, we argue for mechanism-aware evaluation that combines task-relevant symbolic rules with mechanistic interpretability, yielding algorithmic pass/fail scores that show exactly where models generalize versus exploit patterns. We demonstrate this on NL-to-SQL by training two identical architectures under different conditions: one without schema information (forcing memorization), one with schema (enabling grounding). Standard evaluation shows the memorization model achieves 94% field-name accuracy on unseen data, falsely suggesting competence. Our symbolic-mechanistic evaluation reveals this model violates core schema generalization rules, a failure invisible to accuracy metrics.

Related papers

• Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning [28.628342735283752]
  Machine unlearning aims to remove the influence of selected training examples without full retraining.<n>Standard evaluations often summarize unlearning quality with aggregate metrics, such as accuracy- and forgetting-based scores.<n>We study this failure mode at the example level by comparing the predictions of an unlearned model to those of the model retrained after deletion.
  arXiv  Detail & Related papers  (2026-05-29T13:48:06Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Draw a Portrait of Your Graph Data: An Instance-Level Profiling Framework for Graph-Structured Data [3.912899665210116]
  We introduce NodePro, a node profiling framework that enables fine-grained diagnosis of model behavior.<n>We show that node profiles generalize to unseen nodes, supporting prediction reliability without ground-truth labels.<n>We demonstrate the utility of NodePro in identifying semantically inconsistent or corrupted nodes in a structured knowledge graph.
  arXiv  Detail & Related papers  (2025-09-15T16:18:54Z)
• Are We Truly Forgetting? A Critical Re-examination of Machine Unlearning Evaluation Protocols [14.961054239793356]
  We introduce a rigorous unlearning evaluation setup, in which forgetting classes exhibit semantic similarity to downstream task classes.<n>We hope our benchmark serves as a standardized protocol for evaluating unlearning algorithms under realistic conditions.
  arXiv  Detail & Related papers  (2025-03-10T07:11:34Z)
• Probably Approximately Precision and Recall Learning [60.00180898830079]
  A key challenge in machine learning is the prevalence of one-sided feedback.<n>We introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels.<n>We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Semi-supervised Learning For Robust Speech Evaluation [30.593420641501968]
  Speech evaluation measures a learners oral proficiency using automatic models. This paper proposes to address such challenges by exploiting semi-supervised pre-training and objective regularization. An anchor model is trained using pseudo labels to predict the correctness of pronunciation.
  arXiv  Detail & Related papers  (2024-09-23T02:11:24Z)
• Memory Consistency Guided Divide-and-Conquer Learning for Generalized Category Discovery [56.172872410834664]
  Generalized category discovery (GCD) aims at addressing a more realistic and challenging setting of semi-supervised learning. We propose a Memory Consistency guided Divide-and-conquer Learning framework (MCDL) Our method outperforms state-of-the-art models by a large margin on both seen and unseen classes of the generic image recognition.
  arXiv  Detail & Related papers  (2024-01-24T09:39:45Z)
• Weak Supervision Performance Evaluation via Partial Identification [46.73061437177238]
  Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels. We present a novel method to address this challenge by framing model evaluation as a partial identification problem. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques.
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors [105.12462629663757]
  In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
  arXiv  Detail & Related papers  (2022-05-25T15:26:48Z)
• Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics [17.677637487977208]
  Modern summarization models generate highly fluent but often factually unreliable outputs. Due to the lack of common benchmarks, metrics attempting to measure the factuality of automatically generated summaries cannot be compared. We devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems.
  arXiv  Detail & Related papers  (2021-04-27T17:28:07Z)
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification [58.03725169462616]
  We show theoretically that over-parametrization is not the only reason for over-confidence. We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting. Perhaps surprisingly, we also show that over-confidence is not always the case.
  arXiv  Detail & Related papers  (2021-02-15T21:38:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.