The Wiola Architecture for Efficient Small Language Models

• URL: http://arxiv.org/abs/2607.01394v1
• Date: Wed, 01 Jul 2026 18:52:56 GMT
• Title: The Wiola Architecture for Efficient Small Language Models
• Abstract summary: We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles.<n>We provide derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral.<n>Wiola is released in four sizes (120M, 360M, 700M, and 1.5B parameters), and is fully compatible with the Hugging Transformers ecosystem.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles, sharing no structural lineage with any existing model family including GPT, LLaMA, Mistral, or Falcon. Wiola introduces five independently novel components: (i) Spiral Rotary Positional Encoding (SRPE), which embeds token positions on a three-dimensional helical manifold combining absolute, relative, and hierarchical positional signals; (ii) Gated Cross-Layer Attention (GCLA), providing each decoder layer with soft cross-attention access to compressed summaries of two preceding layers for inter-layer coherence; (iii) Adaptive Token Merging (ATM), which dynamically merges se mantically redundant adjacent tokens in middle network layers to reduce attention complexity without information loss; (iv) Dual Stream Feed-Forward (DSFF), replacing the conventional MLP with two parallel streams fused by a learned per-dimension gate; and (v) WiolaRMSNorm, a modified normalisation introducing a per-dimension learned offset vector that prevents representation collapse. We provide complete mathematical derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral. Wiola is released in four sizes (120M, 360M, 700M, and 1.5B parameters) and is fully compatible with the HuggingFace Transformers ecosystem, with all 22 architectural unit tests passing.

Related papers

• MambaADv2: Evolving Duality-enhanced State Space Model for Unsupervised Anomaly Detection [108.36437360254605]
  This paper proposes MambaADv2, a framework tailored for multi-class unsupervised anomaly detection.<n>MambaADv2 comprises a pre-trained encoder and a Mamba-inspired decoder, equipped with Duality-enhanced State Space (DSS) modules across multiple scales.<n>The structure of the Hybrid State Space (HSS) block is tailored by following the SSD-based Mamba lineage and incorporating Mamba3-style position-aware state-space modeling.
  arXiv  Detail & Related papers  (2026-06-22T10:14:06Z)
• FiLM-Coordinated Dual-Branch Transformer for Global-Local Dependency Modeling in Language Modeling [4.131782714245991]
  We propose a FiLM-coordinated dual-branch Transformer for language modeling.<n>The two branches represent different dependency views of the same input.<n>Experiments on small-scale language modeling settings show that the proposed structure consistently outperforms same-width single-branch baselines.
  arXiv  Detail & Related papers  (2026-06-19T03:48:49Z)
• UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval [87.12447641908143]
  We propose UniCVR, the first unified zero-shot composed visual retrieval framework.<n>UniCVR jointly addresses all three tasks without any task-specific human-annotated data.<n>In Stage I, we train the MLLM as a compositional query embedder via contrastive learning on a curated dataset of approximately 3.5M samples.<n>In Stage II, we introduce an MLLM-guided dual-level reranking mechanism that applies adaptive budgeted subset scoring to a small number of top-ranked candidates.
  arXiv  Detail & Related papers  (2026-04-22T08:16:50Z)
• Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Prediction [9.96771578356719]
  Existing approaches rely on increasingly complex network architectures to regress depth maps.<n>We propose a multilevel conditional random field (CRF) model built upon the Swin Transformer backbone.<n>Our method achieves state-of-the-art performance, reducing Abs Rel to 0.088 ($-$7.4%) and RMSE to 0.316 ($-$5.4%) on NYU Depth v2, while attaining near-perfect threshold accuracy.
  arXiv  Detail & Related papers  (2026-04-03T07:59:26Z)
• Search Multilayer Perceptron-Based Fusion for Efficient and Accurate Siamese Tracking [3.7727834708902868]
  Multilayer Perception (MLP)-based fusion module enables pixel-level interaction with minimal structural overhead.<n>Differentiable neural architecture search (DNAS) to decouple channel-width optimization from other architectural choices.<n> tracker ranks among the top performers on four general-purpose and three aerial benchmarks.
  arXiv  Detail & Related papers  (2026-03-02T10:30:54Z)
• TLC-Plan: A Two-Level Codebook Based Network for End-to-End Vector Floorplan Generation [19.063941053235567]
  We propose TLC-Plan, a hierarchical generative model that directly synthesizes vectors from spatial input boundaries.<n>TLC-Plan employs a two-level VQ-VAE to encode global layouts as semantically labeled room bounding boxes.<n>Experiments show state-of-the-art performance on RPLAN dataset and leading results on LI dataset.
  arXiv  Detail & Related papers  (2026-02-06T15:36:50Z)
• LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs [52.24096832965001]
  We present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method.<n>The PVC method can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding.<n>Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x.
  arXiv  Detail & Related papers  (2025-11-26T08:11:10Z)
• UniPTMs: The First Unified Multi-type PTM Site Prediction Model via Master-Slave Architecture-Based Multi-Stage Fusion Strategy and Hierarchical Contrastive Loss [7.9318919792252585]
  This study proposes UniPTMs: the first unified framework for multi-type PTM prediction.<n>UniPTMs demonstrates significant performance improvements over state-of-the-art models.<n>To strike a balance between model complexity and performance, we have also developed a lightweight variant named UniPTMs-mini.
  arXiv  Detail & Related papers  (2025-06-05T13:02:43Z)
• AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation [80.33846577924363]
  We present All-Pairs Multi-Field Transforms (AMT), a new network architecture for video framegithub. It is based on two essential designs. First, we build bidirectional volumes for all pairs of pixels, and use the predicted bilateral flows to retrieve correlations. Second, we derive multiple groups of fine-grained flow fields from one pair of updated coarse flows for performing backward warping on the input frames separately.
  arXiv  Detail & Related papers  (2023-04-19T16:18:47Z)
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition [99.349598600887]
  Conformer is the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture. We propose the Squeezeformer model, which consistently outperforms the state-of-the-art ASR models under the same training schemes.
  arXiv  Detail & Related papers  (2022-06-02T06:06:29Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.