Fine-tuning RoBERTa for CVE-to-CWE Classification: A 125M Parameter Model Competitive with LLMs

• URL: http://arxiv.org/abs/2603.14911v1
• Date: Mon, 16 Mar 2026 07:15:57 GMT
• Title: Fine-tuning RoBERTa for CVE-to-CWE Classification: A 125M Parameter Model Competitive with LLMs
• Abstract summary: We construct a large-scale training dataset of 234,770 descriptions with AI-refined CWE labels.<n>On our held-out test set (27,780 samples, 205 CWE classes), the model achieves 87.4% top-1 accuracy.<n>On the external CTI-Bench benchmark (NeurIPS 2024), the model achieves 75.6% strict accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a fine-tuned RoBERTa-base classifier (125M parameters) for mapping Common Vulnerabilities and Exposures (CVE) descriptions to Common Weakness Enumeration (CWE) categories. We construct a large-scale training dataset of 234,770 CVE descriptions with AI-refined CWE labels using Claude Sonnet 4.6, and agreement-filtered evaluation sets where NVD and AI labels agree. On our held-out test set (27,780 samples, 205 CWE classes), the model achieves 87.4% top-1 accuracy and 60.7% Macro F1 -- a +15.5 percentage-point Macro F1 gain over a TF-IDF baseline that already reaches 84.9% top-1, demonstrating the model's advantage on rare weakness categories. On the external CTI-Bench benchmark (NeurIPS 2024), the model achieves 75.6% strict accuracy (95% CI: 72.8-78.2%) -- statistically indistinguishable from Cisco Foundation-Sec-8B-Reasoning (75.3%, 8B parameters) at 64x fewer parameters. We release the dataset, model, and training code.

Related papers

• Distilling Foundation Models for Agentic What-If Reasoning:Cost, Latency, and Governance in a Hybrid LLM+SLM Architecture [4.205733518246009]
  Tabular foundation models deliver strong zero-training predictive performance via in-context learning, but their high inference latency makes them impractical as hot-path decision backends in interactive agentic loops.<n>We distill a TabPFN teacher into a compact feed-forward student across a business-decision simulation on UCI Adult and five OpenML benchmarks.<n>The student retains 95.4-100.5% accuracy and 96.8-100.0% AUC, with the lowest accuracy retention on credit-g at 95.4%.
  arXiv  Detail & Related papers  (2026-09-14T11:26:21Z)
• Interpretable machine learning for predicting splitting strength of asphalt concrete: insights from SHAP analysis [2.5274533414341493]
  This paper presents an interpretable machine-learning framework for predicting the splitting strength (ST) of asphalt concrete.<n>Six machine-learning models, namely TabPFN, ANN, SVR, RF, XGBoost, and LightGBM, were developed and compared.<n>Results show that TabPFN delivered the best overall performance on the testing set, with the lowest RMSE of 0.28, MAE of 0.21, MAPE of 18.01%, MAD of 0.14, the highest R2 of 0.88, and the highest composite score of 0.91.
  arXiv  Detail & Related papers  (2026-08-02T03:05:54Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• EdgeJury: Cross-Reviewed Small-Model Ensembles for Truthful Question Answering on Serverless Edge Inference [0.0]
  We present EdgeJury, a lightweight ensemble framework that improves truthfulness and robustness.<n>On TruthfulQA (MC1), EdgeJury achieves 76.2% accuracy.<n>On a 200-question adversarial EdgeCases set, EdgeJury yields +48.2% relative gains.
  arXiv  Detail & Related papers  (2025-12-29T14:48:40Z)
• Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures [87.75098311090642]
  Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed.<n>We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres.
  arXiv  Detail & Related papers  (2025-10-16T12:23:13Z)
• Exploring the Hierarchical Reasoning Model for Small Natural-Image Classification Without Augmentation [51.56484100374058]
  It is evaluated on MNIST, CIFAR-10, and CIFAR-100 under a deliberately raw regime.<n>It is concluded that, for small-resolution image classification without augmentation, HRM is not competitive with even simple convolutional architectures.
  arXiv  Detail & Related papers  (2025-10-04T01:22:41Z)
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models [194.64264251080454]
  We present GLM-4.5, an open-source Mixture-of-Experts (MoE) large language model with 355B total parameters and 32B activated parameters.<n>Through multi-stage training on 23T tokens and comprehensive post-training with expert model iteration and reinforcement learning, GLM-4.5 achieves strong performance across agentic, reasoning, and coding tasks.<n>We release both GLM-4.5 (355B parameters) and a compact version, GLM-4.5-Air (106B parameters), to advance research in reasoning and agentic AI systems.
  arXiv  Detail & Related papers  (2025-08-08T17:21:06Z)
• SLEEPYLAND: trust begins with fair evaluation of automatic sleep staging models [0.0]
  We present SLEEPYLAND, an open-source sleep staging evaluation framework.<n>It includes more than 220'000 hours in-domain (ID) sleep recordings, and more than 84'000 hours out-of-domain (OOD) sleep recordings.<n>We introduce SOMNUS, an ensemble combining models across architectures and channel setups via soft voting.
  arXiv  Detail & Related papers  (2025-06-10T08:46:19Z)
• Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning [15.776175440446414]
  We introduce Dr.SoW (Density Ratio of Strong over Weak) a cost-effective method that eliminates the reliance for human annotation.<n>Dr.SoW uses the log-density ratio between a better-aligned and a less-aligned LLM as a reward signal.<n>We preference-tune Llama-3-8B-Instruct using data annotated by Dr.SoW.
  arXiv  Detail & Related papers  (2024-11-04T18:54:39Z)
• Stacking-Enhanced Bagging Ensemble Learning for Breast Cancer Classification with CNN [0.24578723416255752]
  This paper proposes a CNN classification network based on Bagging and stacking ensemble learning methods for breast cancer classification. The model is capable of fast and accurate classification of input images. For binary classification (presence or absence of breast cancer), the accuracy reached 98.84%, and for five-class classification, the accuracy reached 98.34%.
  arXiv  Detail & Related papers  (2024-07-15T09:44:43Z)
• Uncertainty-inspired Open Set Learning for Retinal Anomaly Identification [71.06194656633447]
  We establish an uncertainty-inspired open-set (UIOS) model, which was trained with fundus images of 9 retinal conditions. Our UIOS model with thresholding strategy achieved an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set. UIOS correctly predicted high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images.
  arXiv  Detail & Related papers  (2023-04-08T10:47:41Z)
• Co-supervised learning paradigm with conditional generative adversarial networks for sample-efficient classification [8.27719348049333]
  This paper introduces a sample-efficient co-supervised learning paradigm (SEC-CGAN) SEC-CGAN is trained alongside the classifier and supplements semantics-conditioned, confidence-aware synthesized examples to the annotated data during the training process. Experiments demonstrate that the proposed SEC-CGAN outperforms the external classifier GAN and a baseline ResNet-18 classifier.
  arXiv  Detail & Related papers  (2022-12-27T19:24:31Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.