Test Case Selection for Deep Neural Networks: A Replication Study on LLMs for Code

• URL: http://arxiv.org/abs/2606.27601v1
• Date: Thu, 25 Jun 2026 23:25:53 GMT
• Title: Test Case Selection for Deep Neural Networks: A Replication Study on LLMs for Code
• Abstract summary: Test case selection (TCS) techniques have been explored to support the operational evaluation of deep neural networks (DNNs)<n>We re-examine established TCS strategies originally proposed for DNNs and complement them with statistical sampling strategies not previously evaluated for TCS.<n>We assess their effectiveness on three code-related classification tasks: clone detection, vulnerability detection, and technical debt prediction.
• Score: 12.387676601792897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, test case selection (TCS) techniques have been explored to support the operational evaluation of deep neural networks (DNNs) under limited testing budgets, where labeling cost is a primary concern and uncovering model failures early is a key objective. Although prior studies report promising results, existing empirical evaluations focus almost exclusively on vision-based DNNs and datasets, leaving it unclear whether prior findings generalize to LLM code models. This paper presents a large-scale replication study of TCS techniques in the context of LLM code models. We re-examine established TCS strategies originally proposed for DNNs and complement them with statistical sampling strategies not previously evaluated for TCS. We assess their effectiveness on three code-related classification tasks: clone detection, vulnerability detection, and technical debt prediction. The study spans 17 task-specific fine-tuned model instances, 7 predictive features, and 13 selection strategies, including 12 feature-aware strategies and simple random sampling (SRS) as a feature-agnostic baseline. We evaluate performance along two dimensions: accuracy estimation and early failure discovery. The results indicate that only a subset of findings reported for vision-based DNNs generalize when TCS is applied to LLMs for code. In particular, uncertainty-based features are effective for early failure discovery, while representation-based features are more robust for accuracy estimation. At the same time, performance varies substantially across tasks and models, indicating that TCS effectiveness is context-dependent. Overall, this study provides empirical evidence on the replicability of TCS techniques beyond vision-based deep learning and offers insights into their use for the operational evaluation of LLMs for code.

Related papers

• A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models [6.336623115095147]
  Recent success in natural language processing has motivated growing interest in large-scale foundation models for neuroimaging data.<n>The impact of different tokenization strategies for neural data is currently poorly understood.<n>We present a systematic evaluation of sample-level tokenization strategies for transformer-based large neuroimaging models applied to magnetoencephalography (MEG) data.
  arXiv  Detail & Related papers  (2026-02-18T17:21:02Z)
• Analyzing VLM-Based Approaches for Anomaly Classification and Segmentation [0.0]
  Vision-Language Models (VLMs) have revolutionized anomaly detection by enabling zero-shot and few-shot defect identification without extensive labeled datasets.<n>This project presents a comprehensive analysis of VLM-based approaches for anomaly classification (AC) and anomaly segmentation (AS)
  arXiv  Detail & Related papers  (2026-01-19T22:55:30Z)
• Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation [192.53529928861818]
  Learning with high-resource data has demonstrated substantial success in artificial intelligence (AI)<n>However, the costs associated with data annotation and model training remain significant.<n>This survey employs active sampling theory to analyze the generalization error and label complexity associated with learning from low-resource data.
  arXiv  Detail & Related papers  (2025-10-10T03:15:42Z)
• Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models [64.02612380298228]
  Recent studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT)<n>We propose a model in which TTT achieves a substantially smaller in-distribution test error than global training.<n>We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet.
  arXiv  Detail & Related papers  (2025-09-29T09:24:52Z)
• Mitigating Strategy-Selection Bias in Reasoning for More Effective Test-Time Scaling [27.616118519120366]
  Test-time scaling (TTS) has been shown to improve the performance of large language models (LLMs) by sampling and aggregating diverse reasoning paths.<n>We present a theoretical analysis that reveals when it undermines the effectiveness of test-time scaling.<n>Motivated by this theoretical insight, we introduce TTS-Uniform, a framework designed to mitigate the selection bias of reasoning strategies.
  arXiv  Detail & Related papers  (2025-09-22T15:30:56Z)
• Test-time Offline Reinforcement Learning on Goal-related Experience [50.94457794664909]
  Research in foundation models has shown that performance can be substantially improved through test-time training.<n>We propose a novel self-supervised data selection criterion, which selects transitions from an offline dataset according to their relevance to the current state.<n>Our goal-conditioned test-time training (GC-TTT) algorithm applies this routine in a receding-horizon fashion during evaluation, adapting the policy to the current trajectory as it is being rolled out.
  arXiv  Detail & Related papers  (2025-07-24T21:11:39Z)
• Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMs [1.6332728502735252]
  Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly deployed in industry applications.<n>Their reliability remains hampered by challenges in detecting hallucinations.<n>This paper addresses the bottleneck of data annotation by investigating the feasibility of reducing training data requirements.
  arXiv  Detail & Related papers  (2025-05-29T09:50:56Z)
• Meta-Statistical Learning: Supervised Learning of Statistical Inference [59.463430294611626]
  This work demonstrates that the tools and principles driving the success of large language models (LLMs) can be repurposed to tackle distribution-level tasks.<n>We propose meta-statistical learning, a framework inspired by multi-instance learning that reformulates statistical inference tasks as supervised learning problems.
  arXiv  Detail & Related papers  (2025-02-17T18:04:39Z)
• Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples [53.95282502030541]
  Neural Network-based active learning (NAL) is a cost-effective data selection technique that utilizes neural networks to select and train on a small subset of samples. We try to move one step forward by offering a unified explanation for the success of both query criteria-based NAL from a feature learning view.
  arXiv  Detail & Related papers  (2024-06-06T10:38:01Z)
• Continual Learning in Recurrent Neural Networks [67.05499844830231]
  We evaluate the effectiveness of continual learning methods for processing sequential data with recurrent neural networks (RNNs) We shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs. We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements.
  arXiv  Detail & Related papers  (2020-06-22T10:05:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.