On-device System of Compositional Multi-tasking in Large Language Models

• URL: http://arxiv.org/abs/2510.13848v1
• Date: Sat, 11 Oct 2025 19:49:22 GMT
• Title: On-device System of Compositional Multi-tasking in Large Language Models
• Authors: Ondrej Bohdal, Konstantinos Theodosiadis, Asterios Mpatziakas, Dimitris Filippidis, Iro Spyrou, Christos Zonios, Anastasios Drosou, Dimosthenis Ioannidis, Kyeng-Hun Lee, Jijoong Moon, Hyeonmok Ko, Mete Ozay, Umberto Michieli,
• Abstract summary: We propose a novel approach tailored specifically for compositional multi-tasking scenarios involving summarization and translation.<n>Our technique involves adding a learnable projection layer on top of the combined summarization and translation adapters.<n>We demonstrate the practical viability of our method within an on-device environment by developing an Android app capable of executing compositional tasks seamlessly.
• Score: 29.561801948704822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are commonly adapted for diverse downstream tasks via parameter-efficient fine-tuning techniques such as Low-Rank Adapters (LoRA). While adapters can be combined to handle multiple tasks separately, standard approaches struggle when targeting the simultaneous execution of complex tasks, such as generating a translated summary from a long conversation. To address this challenge, we propose a novel approach tailored specifically for compositional multi-tasking scenarios involving summarization and translation. Our technique involves adding a learnable projection layer on top of the combined summarization and translation adapters. This design enables effective integration while maintaining efficiency through reduced computational overhead compared to alternative strategies requiring extensive retraining or sequential processing. We demonstrate the practical viability of our method within an on-device environment by developing an Android app capable of executing compositional tasks seamlessly. Experimental results indicate our solution performs well and is fast in both cloud-based and on-device implementations, highlighting the potential benefits of adopting our framework in real-world applications demanding high-speed operation alongside resource constraints.

Related papers

• Data-driven Clustering and Merging of Adapters for On-device Large Language Models [34.58968471192321]
  On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks.<n>This raises a critical challenge: how to select representative adapters that generalize well across multiple tasks.<n>We propose a novel method D2C for adapter clustering that leverages minimal task-specific examples and employs an iterative optimization process to refine cluster assignments.
  arXiv  Detail & Related papers  (2026-01-24T12:25:46Z)
• ContextNav: Towards Agentic Multimodal In-Context Learning [85.05420047017513]
  ContextNav is an agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation.<n>It builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts.<n> Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets.
  arXiv  Detail & Related papers  (2025-10-06T07:49:52Z)
• Efficient Compositional Multi-tasking for On-device Large Language Models [34.95345867937008]
  We study the problem of text-based compositional multi-tasking, where each test example involves the simultaneous execution of multiple tasks.<n>Our contributions lay the groundwork for advancing the capabilities of large language models in real-world multi-tasking scenarios, expanding their applicability to complex, resource-constrained use cases.
  arXiv  Detail & Related papers  (2025-07-21T21:39:23Z)
• EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models [64.70546873396624]
  We present the Extremely Complex Instruction Following Benchmark (EIFBENCH) for evaluating large language models (LLMs)<n>EIFBENCH includes multi-task scenarios that enable comprehensive assessment across diverse task types concurrently.<n>We also propose the Segment Policy Optimization (SegPO) algorithm to enhance the LLM's ability to accurately fulfill multi-task workflow.
  arXiv  Detail & Related papers  (2025-06-10T02:39:55Z)
• Efficient Multi-modal Long Context Learning for Training-free Adaptation [96.21248144937627]
  This paper introduces Efficient Multi-Modal Long Context Learning (EMLoC)<n>It embeds demonstration examples directly into the model input.<n>It condenses long-context multimodal inputs into compact, task-specific memory representations.
  arXiv  Detail & Related papers  (2025-05-26T10:49:44Z)
• Pilot: Building the Federated Multimodal Instruction Tuning Framework [79.56362403673354]
  Our framework integrates two stages of "adapter on adapter" into the connector of the vision encoder and the LLM.<n>In stage 1, we extract task-specific features and client-specific features from visual information.<n>In stage 2, we build the cross-task Mixture-of-Adapters(CT-MoA) module to perform cross-task interaction.
  arXiv  Detail & Related papers  (2025-01-23T07:49:24Z)
• MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts [6.245113492272563]
  Mixture of Dyadic Experts (MoDE) is a novel design for efficient multi-task adaptation. Our design allows for more fine-grained mixing, thereby increasing the model's ability to jointly handle multiple tasks.
  arXiv  Detail & Related papers  (2024-08-02T18:05:10Z)
• Task Indicating Transformer for Task-conditional Dense Predictions [16.92067246179703]
  We introduce a novel task-conditional framework called Task Indicating Transformer (TIT) to tackle this challenge. Our approach designs a Mix Task Adapter module within the transformer block, which incorporates a Task Indicating Matrix through matrix decomposition. We also propose a Task Gate Decoder module that harnesses a Task Indicating Vector and gating mechanism to facilitate adaptive multi-scale feature refinement.
  arXiv  Detail & Related papers  (2024-03-01T07:06:57Z)
• Making Small Language Models Better Multi-task Learners with Mixture-of-Task-Adapters [13.6682552098234]
  Large Language Models (LLMs) have achieved amazing zero-shot learning performance over a variety of Natural Language Processing (NLP) tasks. We present ALTER, a system that effectively builds the multi-tAsk learners with mixTure-of-task-adaptERs upon small language models. A two-stage training method is proposed to optimize the collaboration between adapters at a small computational cost.
  arXiv  Detail & Related papers  (2023-09-20T03:39:56Z)
• Multi-Task Learning with Sequence-Conditioned Transporter Networks [67.57293592529517]
  We aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. We propose a new suite of benchmark aimed at compositional tasks, MultiRavens, which allows defining custom task combinations. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling.
  arXiv  Detail & Related papers  (2021-09-15T21:19:11Z)
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections [96.64246471034195]
  We propose textscHyperGrid, a new approach for highly effective multi-task learning. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.
  arXiv  Detail & Related papers  (2020-07-12T02:49:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.