Less Is More: Engineering Challenges of On-Device Small Language Model Integration in a Mobile Application

• URL: http://arxiv.org/abs/2604.24636v2
• Date: Wed, 06 May 2026 08:53:50 GMT
• Title: Less Is More: Engineering Challenges of On-Device Small Language Model Integration in a Mobile Application
• Abstract summary: On-device Small Language Models (SLMs) promise fully offline, private AI experiences for mobile users (no cloud dependency, no data leaving the device).<n>This paper presents a longitudinal practitioner case study documenting the engineering challenges of integrating SLMs into Palabrita, a production Android word-guessing game.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-device Small Language Models (SLMs) promise fully offline, private AI experiences for mobile users (no cloud dependency, no data leaving the device). But is this promise achievable in practice? This paper presents a longitudinal practitioner case study documenting the engineering challenges of integrating SLMs (Gemma 4 E2B, 2.6B parameters; Qwen3 0.6B, 600M parameters) into Palabrita, a production Android word-guessing game. Over a 5-day development sprint comprising 204 commits (~90 directly AI-related), the system underwent a radical transformation: from an ambitious design where the LLM generated complete structured puzzles (word, category, difficulty, and five hints as JSON) to a pragmatic architecture where curated word lists provide the words and the LLM generates only three short hints, with a deterministic fallback if it fails. We identify five categories of failures specific to on-device SLM integration: output format violations, constraint violations, context quality degradation, latency incompatibility, and model selection instability. For each failure category, we document the observed symptoms, root causes, and the prompt engineering and architectural strategies that effectively mitigated them, including multi-layer defensive parsing, contextual retry with failure feedback, session rotation, progressive prompt hardening, and systematic responsibility reduction. Our findings demonstrate that on-device SLMs are viable for production mobile applications, but only when the developer accepts a fundamental constraint: the most reliable on-device LLM feature is one where the LLM does the least. We distill our experience into eight actionable design heuristics for practitioners integrating SLMs into mobile apps.

Related papers

• MeMo: Memory as a Model [96.54179792557863]
  Large language models (LLMs) achieve strong performance across a wide range of tasks, but remain frozen after pretraining until subsequent updates.<n>MeMo is a modular framework that encodes new knowledge into a dedicated memory model while keeping the LLM parameters unchanged.<n>MeMo achieves strong performance compared to existing methods across diverse settings.
  arXiv  Detail & Related papers  (2026-05-14T17:51:34Z)
• Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots [15.63408997133083]
  We introduce DailyDroid, a benchmark of 75 tasks in five scenarios across 25 Android apps, spanning three difficulty levels to mimic everyday smartphone use.<n>We evaluate it using text-only and multimodal (text + screenshot) inputs on GPT-4o and o4-mini across 300 trials, revealing comparable performance with multimodal inputs yielding marginally higher success rates.
  arXiv  Detail & Related papers  (2026-04-20T05:15:14Z)
• Review of Tools for Zero-Code LLM Based Application Development [0.6978180153516672]
  Large Language Models (LLMs) are transforming software creation by enabling zero code development platforms.<n>Our survey reviews recent platforms that let users build applications without writing code, by leveraging LLMs as the brains of the development process.
  arXiv  Detail & Related papers  (2025-10-22T16:41:16Z)
• Semantic-Aware Fuzzing: An Empirical Framework for LLM-Guided, Reasoning-Driven Input Mutation [0.5336076422485075]
  Security vulnerabilities in Internet-of-Things devices, mobile platforms, and autonomous systems remain critical.<n>Traditional mutation-based fuzzers primarily perform byte or bit-level edits without semantic reasoning.<n>We present an open-source framework that integrates reasoning LLMs with AFL++ on Google's FuzzBench.
  arXiv  Detail & Related papers  (2025-09-23T19:57:29Z)
• LLM Applications: Current Paradigms and the Next Frontier [8.214897650566494]
  The development of large language models (LLMs) has given rise to four major application paradigms.<n>Each has its advantages but also shares common challenges.<n>This paper reviews and analyzes these paradigms, covering architecture design, application ecosystem, research progress, as well as the challenges and open problems they face.
  arXiv  Detail & Related papers  (2025-03-06T16:38:23Z)
• LAMD: Context-driven Android Malware Detection and Classification with LLMs [8.582859303611881]
  Large Language Models (LLMs) offer a promising alternative with their zero-shot inference and reasoning capabilities.<n>We propose LAMD, a practical context-driven framework to enable LLM-based Android malware detection.
  arXiv  Detail & Related papers  (2025-02-18T17:01:37Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints [86.59857711385833]
  We introduce RealInstruct, the first benchmark designed to evaluate LLMs' ability to follow real-world multi-constrained instructions. To address the performance gap between open-source and proprietary models, we propose the Decompose, Critique and Refine (DeCRIM) self-correction pipeline. Our results show that DeCRIM improves Mistral's performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
  arXiv  Detail & Related papers  (2024-10-09T01:25:10Z)
• PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion [96.47420221442397]
  We construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. We test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates robustness settings. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark.
  arXiv  Detail & Related papers  (2024-03-06T15:33:32Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.