EdgeFM: Efficient Edge Inference for Vision-Language Models

• URL: http://arxiv.org/abs/2604.27476v1
• Date: Thu, 30 Apr 2026 06:18:50 GMT
• Title: EdgeFM: Efficient Edge Inference for Vision-Language Models
• Abstract summary: We propose EdgeFM, a lightweight, agent-driven VLM/LLM inference framework tailored for cross-platform industrial edge deployment.<n>By allowing direct invocation of skills, it effectively closes the performance gap long dominated by proprietary toolchains.<n>We show that EdgeFM delivers favorable end-to-end inference performance, providing an open-source, production-grade solution for diverse edge industrial scenarios.
• Score: 11.086903401744713
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) have demonstrated strong applicability in edge industrial applications, yet their deployment remains severely constrained by requirements for deterministic low latency and stable execution under resource limitations. Existing frameworks either rely on bloated general-purpose designs or force developers into opaque, hardware-specific closed-source ecosystems, leading to hardware lock-in limitation and poor cross-platform adaptability. Observing that modern AI agents can efficiently search and tune configurations to generate highly optimized low-level kernels for standard LLM operators, we propose EdgeFM, a lightweight, agent-driven VLM/LLM inference framework tailored for cross-platform industrial edge deployment. EdgeFM removes non-essential features to reduce single-request latency, and encapsulates agent-tuned kernel optimizations as a modular library of reusable skills. By allowing direct invocation of these skills rather than waiting for closed-source implementations, it effectively closes the performance gap long dominated by proprietary toolchains. The framework natively supports mainstream platforms including x86 and NVIDIA Orin SoCs, and represents the first end-to-end VLA deployment on the domestic Horizon Journey platform, enhancing cross-platform portability. In most cases, it yields clearly better inference performance than conventional vendor-specific toolchains, achieving up to 1.49 times speedup over TensorRT-Edge-LLM on the NVIDIA Orin platform. Experimental results show that EdgeFM delivers favorable end-to-end inference performance, providing an open-source, production-grade solution for diverse edge industrial scenarios.

Related papers

• AgenTEE: Confidential LLM Agent Execution on Edge Devices [4.9434226170553535]
  AgenTEE is a system for deploying confidential agent pipelines on edge devices.<n>It enforces strong system-level isolation of sensitive assets and runtime state.<n>Our evaluation shows that such multi-cVMs system is practical, achieving near-native performance with less than 5.15% runtime overhead.
  arXiv  Detail & Related papers  (2026-04-20T13:13:31Z)
• Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends [0.2578242050187029]
  Edge devices typically operate under strict constraints on processing power, memory, and energy consumption.<n>Small Language Models (SLMs) offer lightweight alternatives that bring AI inference to resource-constrained environments.<n>We analyze both maximum achievable performance and processing and energy efficiency across commercial solutions available for each platform.
  arXiv  Detail & Related papers  (2025-11-27T11:11:01Z)
• AIvailable: A Software-Defined Architecture for LLM-as-a-Service on Heterogeneous and Legacy GPUs [0.5863360388454261]
  We introduce AIvailable, a low-cost, highly available LLM-as-a-Service (LLM) platform.<n>It uses a software-defined approach for running LLMs across heterogeneous and legacy GPU nodes.<n>It features a unified client interface that allows seamless interaction with all deployed LLMs.
  arXiv  Detail & Related papers  (2025-11-06T14:19:57Z)
• MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning [91.90342432541138]
  Scaling up model size and training data has advanced foundation models for instance-level perception.<n>High computational cost limits adoption on resource-constrained platforms.<n>We introduce a new benchmark for efficient segmentation on both high-performance computing platforms and mobile devices.
  arXiv  Detail & Related papers  (2025-10-16T18:00:00Z)
• xLLM Technical Report [57.13120905321185]
  We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework.<n>xLLM builds a novel decoupled service-engine architecture.<n>xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources.
  arXiv  Detail & Related papers  (2025-10-16T13:53:47Z)
• Edge-First Language Model Inference: Models, Metrics, and Tradeoffs [0.7980273012483663]
  This work examines the interplay between edge and cloud deployments, starting from detailed benchmarking of SLM capabilities on single edge devices.<n>We identify scenarios where edge inference offers comparable performance with lower costs, and others where cloud fallback becomes essential due to limits in scalability or model capacity.<n>Rather than proposing a one-size-fits-all solution, we present platform-level comparisons and design insights for building efficient, adaptive LM inference systems.
  arXiv  Detail & Related papers  (2025-05-22T10:43:00Z)
• The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networks [56.37880529653111]
  The demand for large computation model (LAIM) services is driving a paradigm shift from traditional cloud-based inference to edge-based inference for low-latency, privacy-preserving applications.<n>In this paper, we investigate the LAIM-inference scheme, where a pre-trained LAIM is pruned and partitioned into on-device and on-server sub-models for deployment.
  arXiv  Detail & Related papers  (2025-05-14T08:18:55Z)
• WDMoE: Wireless Distributed Mixture of Experts for Large Language Models [68.45482959423323]
  Large Language Models (LLMs) have achieved significant success in various natural language processing tasks. We propose a wireless distributed Mixture of Experts (WDMoE) architecture to enable collaborative deployment of LLMs across edge servers at the base station (BS) and mobile devices in wireless networks.
  arXiv  Detail & Related papers  (2024-11-11T02:48:00Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• Bridge the Future: High-Performance Networks in Confidential VMs without Trusted I/O devices [9.554247218443939]
  Trusted I/O (TIO) is an appealing solution to improve I/O performance for confidential impact (CVMs) This paper emphasizes that not all types of I/O can derive substantial benefits from TIO, particularly network I/O. We present FOlio, a software solution crafted from a secure and efficient Data Plane Development Kit (DPDK) extension.
  arXiv  Detail & Related papers  (2024-03-05T23:06:34Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.