Alteron: A Tool for Behavioral Regression Testing Across NLP Classifier Versions

• URL: http://arxiv.org/abs/2607.29557v1
• Date: Fri, 31 Jul 2026 15:48:31 GMT
• Title: Alteron: A Tool for Behavioral Regression Testing Across NLP Classifier Versions
• Abstract summary: evaluating evolving Natural Language Processing (NLP) models is important for ensuring reliable behavior across updates.<n>Standard benchmark metrics do not fully capture how model behavior changes across versions.<n>We present Alteron, a tool for detecting behavioral regressions across NLP model versions with metamorphic testing.
• Score: 0.3823356975862005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating evolving Natural Language Processing (NLP) models is important for ensuring reliable behavior across updates, but standard benchmark metrics do not fully capture how model behavior changes across versions. Existing work has focused mainly on testing models in isolation rather than comparing successive versions in continuous integration workflows. We present Alteron, a tool for detecting behavioral regressions across NLP model versions with metamorphic testing. Alteron constructs a test corpus from labeled source examples and compares model versions on metamorphically transformed inputs. In an evaluation spanning 10 metamorphic relations (MRs), 4 model versions, and 3 model-update transitions, Alteron identified 16 behavioral regressions, 11 of which were release-blocking. The results show that common model updates can preserve overall task performance while still introducing undesirable behavior changes, and that behavioral checks across model versions can reveal failures that aggregate benchmark metrics alone do not capture. The tool is open-source and available at https://github.com/shazzad5709/alteron. A screencast demonstration is available at https://youtu.be/szwiWW5O4do.

Related papers

• AutoStan: Autonomous Bayesian Model Improvement via Predictive Feedback [0.38073142980733005]
  AutoStan is a framework in which a command-line interface (CLI) coding agent autonomously builds and iteratively improves Bayesian models written in Stan.<n>We evaluate AutoStan on five datasets with diverse modeling structures.
  arXiv  Detail & Related papers  (2026-03-29T16:58:46Z)
• Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens [53.99177152562075]
  Scaling up autoregressive models in vision has not proven as beneficial as in large language models. We focus on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed order using BERT- or GPT-like transformer architectures. Our results show that while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends.
  arXiv  Detail & Related papers  (2024-10-17T17:59:59Z)
• RS-Reg: Probabilistic and Robust Certified Regression Through Randomized Smoothing [19.03441416869426]
  We show how to establish upper bounds on input data point using the $ell$ norm. We then derive a certified upper bound of the perturbation inputs when dealing with a family of regression models where the outputs are bounded. Our simulations verify the validity of the theoretical results and reveal the advantages and limitations of simple smoothing functions.
  arXiv  Detail & Related papers  (2024-05-14T18:10:46Z)
• Importance of Disjoint Sampling in Conventional and Transformer Models for Hyperspectral Image Classification [2.1223532600703385]
  This paper presents an innovative disjoint sampling approach for training SOTA models on Hyperspectral image classification (HSIC) tasks. By separating training, validation, and test data without overlap, the proposed method facilitates a fairer evaluation of how well a model can classify pixels it was not exposed to during training or validation. This rigorous methodology is critical for advancing SOTA models and their real-world application to large-scale land mapping with Hyperspectral sensors.
  arXiv  Detail & Related papers  (2024-04-23T11:40:52Z)
• Post-Hoc Reversal: Are We Selecting Models Prematurely? [13.910702424593797]
  We show a phenomenon that we call post-hoc reversal, where performance trends are reversed after applying post-hoc transforms. Preliminary analyses suggest that these transforms induce reversal by suppressing the influence of mislabeled examples. We propose post-hoc selection, a simple technique whereby post-hoc metrics inform model development decisions.
  arXiv  Detail & Related papers  (2024-04-11T14:58:19Z)
• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data [49.73114504515852]
  We show that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse. We demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse.
  arXiv  Detail & Related papers  (2024-04-01T18:31:24Z)
• On-the-Fly Test-time Adaptation for Medical Image Segmentation [63.476899335138164]
  Adapting the source model to target data distribution at test-time is an efficient solution for the data-shift problem. We propose a new framework called Adaptive UNet where each convolutional block is equipped with an adaptive batch normalization layer. During test-time, the model takes in just the new test image and generates a domain code to adapt the features of source model according to the test data.
  arXiv  Detail & Related papers  (2022-03-10T18:51:29Z)
• Predicting Out-of-Distribution Error with the Projection Norm [87.61489137914693]
  Projection Norm predicts a model's performance on out-of-distribution data without access to ground truth labels. We find that Projection Norm is the only approach that achieves non-trivial detection performance on adversarial examples.
  arXiv  Detail & Related papers  (2022-02-11T18:58:21Z)
• Comparing Test Sets with Item Response Theory [53.755064720563]
  We evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.
  arXiv  Detail & Related papers  (2021-06-01T22:33:53Z)
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates [68.09049111171862]
  This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. We formulate the regression-free model updates into a constrained optimization problem. We empirically analyze how model ensemble reduces regression.
  arXiv  Detail & Related papers  (2021-05-07T03:33:00Z)
• Detection of data drift and outliers affecting machine learning model performance over time [5.319802998033767]
  Drift is distribution change between the training and deployment data. We wish to detect these changes but can't measure accuracy without deployment data labels. We instead detect drift indirectly by nonparametrically testing the distribution of model prediction confidence for changes.
  arXiv  Detail & Related papers  (2020-12-16T20:50:12Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.