From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT

• URL: http://arxiv.org/abs/2605.03686v2
• Date: Wed, 06 May 2026 11:06:46 GMT
• Title: From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT
• Abstract summary: We present a classification task integrated into the NNGPT framework.<n>A fine-tuned LLM predicts which of two image classification datasets a given neural network architecture achieves higher accuracy on.<n>The results establish that LLMs can be fine-tuned to predict cross-dataset suitability from neural network code.
• Score: 48.83701310501069
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated Machine Learning (AutoML) frameworks increasingly leverage Large Language Models (LLMs) for tasks such as hyperparameter optimization and neural architecture code generation. However, current LLM-based approaches focus on generative outputs and evaluate them by training the produced artifacts. Whether LLMs can learn to reason about neural network performance across datasets remains underexplored. We present a classification task integrated into the NNGPT framework, in which a fine-tuned LLM predicts which of two image classification datasets a given neural network architecture achieves higher accuracy on. The task is built on the LEMUR dataset, which provides standardized PyTorch implementations with reproducible performance metrics. Three prompt configurations of increasing difficulty are evaluated: a normalized-accuracy baseline (trivially reaching 100%), a metadata-enriched prompt replacing accuracies with dataset properties, and a code-only prompt presenting only architecture source code and dataset names. Using DeepSeek-Coder-7B-Instruct fine-tuned with LoRA, the code-only prompt reaches 80% peak accuracy over 15 epochs, while the metadata prompt peaks at 70%. Perdataset analysis reveals complementary strengths: metadata excels for datasets with distinctive properties (CelebAGender at 90.9%) but degrades for overlapping characteristics, whereas the code-only prompt shows more balanced performance. A comparison with DeepSeek-Coder1.3B confirms that model capacity affects this form of architectural reasoning. The results establish that LLMs can be fine-tuned to predict cross-dataset suitability from neural network code, suggesting that architecture source code contains richer discriminative signal than dataset metadata alone.

Related papers

• MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation [51.15583711678273]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation.<n>Within each loop iteration, the MLLM-DataEngine first analyzes the weakness of the model based on the evaluation results.<n>We find MLLM-DataEngine could boost the MLLMs capability in a targeted and automatic manner without human participants.
  arXiv  Detail & Related papers  (2026-07-07T08:35:38Z)
• NNGPT: Rethinking AutoML with Large Language Models [36.90850535125572]
  NNGPT is an open-source framework that turns a large language model (LLM) into a self-improving AutoML engine for neural network development.<n>It integrates within one unified workflow five synergistic LLM-based pipelines: zero-shot architecture synthesis, hyper parameter optimization, code-aware accuracy/early-stop prediction, and reinforcement learning.<n>The system has already generated over 5K validated models, proving NNGPT as an autonomous AutoML engine.
  arXiv  Detail & Related papers  (2025-11-25T14:10:44Z)
• Private Training & Data Generation by Clustering Embeddings [74.00687214400021]
  Differential privacy (DP) provides a robust framework for protecting individual data.<n>We introduce a novel principled method for DP synthetic image embedding generation.<n> Empirically, a simple two-layer neural network trained on synthetically generated embeddings achieves state-of-the-art (SOTA) classification accuracy.
  arXiv  Detail & Related papers  (2025-06-20T00:17:14Z)
• LEMUR Neural Network Dataset: Towards Seamless AutoML [35.57280723615144]
  We introduce LEMUR, an open-source dataset and framework that provides a large collection of PyTorch-based neural networks.<n>Each model follows a unified template, with configurations and results stored in a structured database to ensure consistency.<n>LEMUR aims to accelerate AutoML research, enable fair benchmarking, and reduce barriers to large-scale neural network research.
  arXiv  Detail & Related papers  (2025-04-14T09:08:00Z)
• Embedding Domain-Specific Knowledge from LLMs into the Feature Engineering Pipeline [0.0]
  We propose using Large Language Models (LLMs) as an initial feature construction step to add knowledge to the dataset.<n>Our results show that the evolution can converge faster, saving us computational resources.
  arXiv  Detail & Related papers  (2025-03-27T04:48:58Z)
• SnipGen: A Mining Repository Framework for Evaluating LLMs for Code [51.07471575337676]
  Language Models (LLMs) are trained on extensive datasets that include code repositories.<n> evaluating their effectiveness poses significant challenges due to the potential overlap between the datasets used for training and those employed for evaluation.<n>We introduce SnipGen, a comprehensive repository mining framework designed to leverage prompt engineering across various downstream tasks for code generation.
  arXiv  Detail & Related papers  (2025-02-10T21:28:15Z)
• MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data. For quality, we resort to GPT-4 to generate high-quality data with each given data type.
  arXiv  Detail & Related papers  (2023-08-25T01:41:04Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction [3.3810628880631226]
  This paper systematically investigates the combination of log data embedding strategies and Deep Learning (DL) types for failure prediction. To that end, we propose a modular architecture to accommodate various configurations of embedding strategies and DL-based encoders. Using the F1 score metric, our results show that the best overall performing configuration is a CNN-based encoder with Logkey2vec.
  arXiv  Detail & Related papers  (2023-03-13T16:04:14Z)
• NASirt: AutoML based learning with instance-level complexity information [0.0]
  We present NASirt, an AutoML methodology that finds high accuracy CNN architectures for spectral datasets. Our method performs, in most cases, better than the benchmarks, achieving average accuracy as high as 97.40%.
  arXiv  Detail & Related papers  (2020-08-26T22:21:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.