StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design

• URL: http://arxiv.org/abs/2607.22708v1
• Date: Mon, 20 Jul 2026 14:14:44 GMT
• Title: StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design
• Abstract summary: We present StepX-Edge, a on-device UI vision-language model that resolves tension through three co-grained co-grained architecture, training, and deployment.<n>For training, the five-stage StepX-Curriculum framework is designed around our observation of mutual-promotion effects among UI subtasks.<n>For deployment, a module-wise differentiated two-stage PTQ-to-QAT quantization scheme keeps the post-quantization accuracy loss within 1%.
• Score: 8.502778898017556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deploying a vision-language model with full UI understanding on end devices has long been trapped between accuracy and efficiency: on one side is the accuracy bar for OCR, screen understanding, visual question answering, and element grounding; on the other is the strict compute, memory, and power budget of mobile chips. Existing work either trades one for the other, or stops at simulation without real-device validation. We present StepX-Edge, a 0.9B-parameter on-device UI vision-language model that resolves this tension through three-layer co-design of architecture, training, and deployment. Architecturally, UI-aware Layered Visual Encoding (ULVE) and a Progressive Dimensionality Projection (PDP) connector target the extreme aspect ratios and fine-grained perception of screens, while standard full attention throughout ensures native compatibility with mainstream mobile NPU operators. For training, the five-stage StepX-Curriculum framework is designed around our observation of mutual-promotion effects among UI subtasks, so that all four capabilities grow synergistically under a tight parameter budget rather than interfering. For deployment, a module-wise differentiated two-stage PTQ-to-QAT quantization scheme keeps the post-quantization accuracy loss within 1%. StepX-Edge achieves the strongest overall UI understanding among <=1B models, surpassing all 2B-2.3B baselines on ScreenQA (88.76 F1) and Chinese OCRBench v2 (57.25), and matching 1.3B-2.3B general VLMs on RefCOCO (92.0%) and OCRBench v1 (831) with far fewer parameters. After W4A16+KV8 quantization, the model runs stably on Snapdragon 8 Gen5 devices with ~0.84 s TTFT, 98 tok/s decode, and 1.4 GB peak memory. We will open-source the training data, the full training recipe, and the quantization deployment pipeline.

Related papers

• Opt.Gear Technical Report [3.573729952898132]
  OptGear is a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability.<n>It includes a dense model (1M, 270M, and 1B) with a context length of 64K.<n>OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU.
  arXiv  Detail & Related papers  (2026-08-02T06:43:25Z)
• HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents [57.53434680014667]
  HyMobileAgent is a vision-native foundation model built on Hy3.0-VL-A3B.<n>We develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction.
  arXiv  Detail & Related papers  (2026-07-16T04:12:42Z)
• Joint Architecture-Token-Bitwidth Multi-Axis Optimization of Vision Transformers for Semiconductor IC Packaging [32.78037242946664]
  Vision Transformers (ViTs) have achieved strong performance in visual recognition, yet their deployment in resource-constrained industrial environments remains limited.<n>We present one of the first holistic frameworks that jointly optimize three complementary axes: architecture, token, and bit-width.<n>Specifically, the framework identifies compact backbones via Neural Architecture Search (AutoFormer), reduces information processing via token merging (ToMe), and accelerates per-operation execution via fp16 mixed-precision inference.
  arXiv  Detail & Related papers  (2026-05-03T06:45:24Z)
• Scalable Training of Mixture-of-Experts Models with Megatron Core [26.9162079065285]
  Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models.<n>Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation.<n>We address these challenges for MoE training through integrated optimizations spanning memory (fine-dimensional recomputation, offloading), communication, and computation.
  arXiv  Detail & Related papers  (2026-03-08T15:42:43Z)
• Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs [49.99513618431772]
  We propose a hardware co-design law that captures model accuracy and inference performance.<n>We empirically evaluate 1,942 candidate architectures on NVIDIA Jetson Orin.<n>Our architecture achieves 19.42% lower perplexity on WikiText-2.
  arXiv  Detail & Related papers  (2026-02-10T23:51:00Z)
• UI-Venus-1.5 Technical Report [64.4832043785725]
  We present UI-Venus-1.5, a unified, end-to-end GUI Agent.<n>The proposed model family comprises two dense variants (2B and 8B) and one mixture-of-experts variant (30B-A3B)<n>In addition, UI-Venus-1.5 demonstrates robust navigation capabilities across a variety of Chinese mobile apps.
  arXiv  Detail & Related papers  (2026-02-09T18:43:40Z)
• STEP3-VL-10B Technical Report [115.89015065130127]
  STEP3-VL-10B is a lightweight foundation model designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence.<n>We implement Parallel Coordinated Reasoning (PaCoRe) to scale test-time compute, allocating resources to scalable perceptual reasoning.<n>It records 92.2% on MMBench and 80.11% on MMMU, while excelling in complex reasoning with 94.43% on AIME2025 and 75.95% on MathVision.
  arXiv  Detail & Related papers  (2026-01-14T17:58:24Z)
• Lightweight Transformer Architectures for Edge Devices in Real-Time Applications [0.0]
  This survey examines lightweight transformer architectures specifically designed for edge deployment.<n>We systematically review prominent lightweight variants including MobileBERT, TinyBERT, DistilBERT, EfficientFormer, EdgeFormer, and MobileViT.<n> Experimental results demonstrate that modern lightweight transformers can achieve 75-96% of full-model accuracy while reducing model size by 4-10x and inference latency by 3-9x.
  arXiv  Detail & Related papers  (2026-01-05T01:04:25Z)
• EVCC: Enhanced Vision Transformer-ConvNeXt-CoAtNet Fusion for Classification [0.5394291557377919]
  Hybrid vision architectures combining Transformers and CNNs have significantly advanced image classification, but they usually do so at significant computational cost.<n>We introduce EVCC, a novel multi-branch architecture integrating the Vision Transformer, lightweight ConvNeXt, and CoAtNet.<n> Experiments across the CIFAR-100, Tobacco3482, CelebA, and Brain Cancer datasets demonstrate EVCC's superiority over powerful models.
  arXiv  Detail & Related papers  (2025-11-24T02:11:19Z)
• Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts [60.1586169973792]
  M$3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE) MoE achieves better accuracy and over 80% reduction computation but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations.
  arXiv  Detail & Related papers  (2023-05-30T02:24:03Z)
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications [68.35683849098105]
  We introduce split depth-wise transpose attention (SDTA) encoder that splits input tensors into multiple channel groups. Our EdgeNeXt model with 1.3M parameters achieves 71.2% top-1 accuracy on ImageNet-1K. Our EdgeNeXt model with 5.6M parameters achieves 79.4% top-1 accuracy on ImageNet-1K.
  arXiv  Detail & Related papers  (2022-06-21T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.