VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use

• URL: http://arxiv.org/abs/2605.13989v3
• Date: Thu, 21 May 2026 13:59:18 GMT
• Title: VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use
• Abstract summary: We present VectraYX-Nano, a 41.95M- parameter decoder-only language model trained from scratch in Spanish for cybersecurity.<n>A 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at $25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M)<n> Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/-
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present VectraYX-Nano, a 41.95M-parameter decoder-only language model trained from scratch in Spanish for cybersecurity, with a Latin-American regional focus and native tool invocation via the Model Context Protocol (MCP). The model has four contributions. (i) Corpus: VectraYX-Sec-ES, a 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at ~$25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M). (ii) Architecture: a 42M Transformer decoder with GQA, QK-Norm, RMSNorm, SwiGLU, RoPE and z-loss, paired with a domain-balanced 16,384-token byte-fallback BPE. (iii) Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/- 0.043 on B5 over N=4 seeds, and a controlled Phase-2 replay sweep over {0,5,10,25,50}% saturates B5 at >=25% replay. (iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES), and v6 (60/25/15 OpenSubs/mC4/Wiki) exposes a loss-versus-register inversion: lower-perplexity bootstraps yield measurably worse conversational behavior (v2 > v4 > v6 on B5 at every paired seed). The B4 (tool-selection) floor of 0.000 is a corpus-density artifact, not a capacity gate: rebalancing the SFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, the released headline configuration, reaching B4 = 0.230 +/- 0.052 at 42M while retaining B1 = 0.332 +/- 0.005 and B5 = 0.725 +/- 0.130; a LoRA replication on a 260M from-scratch mid-tier reaches 0.445 +/- 0.201. The released GGUF is 96 MB in F16, runs sub-second TTFT on commodity hardware under llama.cpp, and is, to our knowledge, the first published Spanish-native cybersecurity LLM with end-to-end MCP integration.

Related papers

• UE5M3 FP4 Block Scaling for Stable Language Model Pretraining [6.963061311516306]
  We pretrain a model for nearly 190 billion tokens using NVIDIA's Transformer Engine nv recipe.<n>Compared with Transformer Engine nv, the proposed block-16 recipe finishes with lower final-window training loss and, under their respective quantized-inference policies, lower validation loss measured as held-out negative log-likelihood.<n>A native nv execution ablation that jointly removes RHT and the BF16 final-block exemption increases measured model-body token throughput by 21.2%.
  arXiv  Detail & Related papers  (2026-09-02T17:32:07Z)
• REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs [0.0]
  Most vision-language-action (VLA) models are monolithic.<n>We introduce REFACTOR-VLA, a wake/sleepmodel system for learning reusable skills.
  arXiv  Detail & Related papers  (2026-09-01T13:19:05Z)
• M-QAM MIMO Maximum-Likelihood Detection with QAOA: ML-Rate Offline Angle Design and Correlated Infinite-Size Spin-Glass Models [9.13755431537592]
  The quantum approximate optimization algorithm (QAOA) targets NP-hard maximum-likelihood (ML) detection in multiple-input multiple-output (MIMO) systems.<n>Tests reach 4096-QAM, 128 antennas, $p=30$ and per-symbol SNR 0-45 dB.<n>The approach points toward near-optimum decoding on deeper noiseless fault-tolerant quantum (FTQ) circuits.
  arXiv  Detail & Related papers  (2026-08-18T12:45:47Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Use [0.0]
  We present VectraYX-Vision-1B, a sub-2B vision-language model (VLM) for Spanish/LATAM0: imagery.<n>It answers in Spanish, emits structured reasoning via native |think|> tokens, invokes tools via Model Protocol (|tool_call|>), and exports to llama's LLaVA mmproj format for air-gapped deployment.
  arXiv  Detail & Related papers  (2026-08-09T04:46:58Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models [45.62029693245481]
  We introduce ActQuant, an action-guided mixed-precision PTQ framework.<n>ActQuant operates in two stages: (1) an inter-tensor bit allocator that assigns each weight matrix a single bitwidth based on how much it contributes to predicting the agent's actions.<n>We also introduce OmniModel, an agentic conversion pipeline that ports into a native C/C++ with efficient low-bit kernels.
  arXiv  Detail & Related papers  (2026-05-19T19:57:26Z)
• Measuring Maximum Activations in Open Large Language Models [60.3514350516308]
  We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
  arXiv  Detail & Related papers  (2026-05-15T03:31:51Z)
• The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive [0.0]
  CPU-only scoring primitive runs at 2.6 microseconds per token.<n> token rank-frequency distributions converge to the same two- parameter Mandelbrot ranking distribution.<n>We derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs.
  arXiv  Detail & Related papers  (2026-04-28T13:35:31Z)
• ML Defender (aRGus NDR): An Open-Source Embedded ML NIDS for Botnet and Anomalous Traffic Detection in Resource-Constrained Organizations [0.0]
  We present an open-source network intrusion detection system built in C++20, deployable on commodity hardware at 150-200 USD.<n> ML Defender implements a six-component pipeline over eBPF/XDP packet capture, ZeroMQ transport, and Protocol Buffers serialization.<n>The Ransomware Threat Wins policy selects the maximum arithmetic of both scores, using ML inference to suppress false positives.
  arXiv  Detail & Related papers  (2026-04-03T05:20:13Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development [0.0]
  PashtoCorp is a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains underrepresented in NLP.<n>At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus.
  arXiv  Detail & Related papers  (2026-03-17T10:36:18Z)
• David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation [1.6752458252726459]
  NL-to-Scenic generation with large language models (LLMs) suffers from scarce data, limited metrics.<n>We introduce NL2Scenic, an open dataset and framework with 146 NL/Scenic pairs, a difficulty-stratified 30-case test split, and an Example Retriever.<n>We evaluate 13 models: four proprietary (GPT-4o, GPT-5, Claude-Sonnet-4, Gemini-2.5-pro) and nine open-source code models (Qwen2.5Coder 0.5B-32B; CodeLlama 7B/13B/34B)
  arXiv  Detail & Related papers  (2025-10-15T21:37:02Z)
• ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks [12.396822247035578]
  We present exMT, a benchmark for objective extraction and metacognition.<n>Given a multi-turn transcript, a model must output a one-sentence base objective and a self-reported confidence.<n> Accuracy is scored by similarity to gold objectives, then thresholded once on 300 calibration items.
  arXiv  Detail & Related papers  (2025-08-23T03:32:04Z)
• Vuyko Mistral: Adapting LLMs for Low-Resource Dialectal Translation [0.0]
  This paper introduces the first effort to adapt large language models to the Ukrainian dialect Hutsul.<n>We created a parallel corpus of 9852 dialect-to-standard Ukrainian sentence pairs and a dictionary of 7320 dialectal word mappings.
  arXiv  Detail & Related papers  (2025-06-09T10:30:35Z)
• Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity [56.0251572416922]
  State Space Models (SSMs) have emerged as efficient alternatives to Transformers for sequential modeling.<n>We propose a novel SSM architecture that introduces modality-aware sparsity through modality-specific parameterization of the Mamba block.<n>We evaluate Mixture-of-Mamba across three multi-modal pretraining settings.
  arXiv  Detail & Related papers  (2025-01-27T18:35:05Z)
• Towards Robust Speech Representation Learning for Thousands of Languages [77.2890285555615]
  Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. We propose XEUS, a Cross-lingual for Universal Speech, trained on over 1 million hours of data across 4057 languages.
  arXiv  Detail & Related papers  (2024-06-30T21:40:26Z)
• Nearest Neighbor Speculative Decoding for LLM Generation and Attribution [87.3259169631789]
  Nearest Speculative Decoding (NEST) is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources.<n>NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks.<n>In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B.
  arXiv  Detail & Related papers  (2024-05-29T17:55:03Z)
• Nonparametric Masked Language Modeling [113.71921977520864]
  Existing language models (LMs) predict tokens with a softmax over a finite vocabulary. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval.
  arXiv  Detail & Related papers  (2022-12-02T18:10:42Z)
• Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks [77.90900650816046]
  We introduce $textZemi$, a zero-shot semi-parametric language model. We train $textZemi$ with a novel semi-parametric multitask prompted training paradigm. Specifically, we augment the multitask training and zero-shot evaluation with retrieval from a large-scale task-agnostic unlabeled corpus.
  arXiv  Detail & Related papers  (2022-10-01T04:08:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.