DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation

• URL: http://arxiv.org/abs/2606.29961v1
• Date: Mon, 29 Jun 2026 08:38:40 GMT
• Title: DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation
• Abstract summary: DuoMem is a dual-space distillation framework that transfers procedural problem-solving ability from a large teacher model to compact student models.<n>It is evaluated on ALFWorld, a challenging embodied decision-making benchmark.<n>The DuoMem-enhanced 4B model completes tasks over 3x faster than the 72B teacher in wall-clock time.
• Score: 26.38620865128242
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM)-based agents can solve complex procedural tasks by interacting with environments over multiple turns, but this ability typically depends on large models, long contexts, and repeated inference calls. This makes advanced memory-augmented agents difficult to deploy on resource-constrained devices. We introduce DuoMem, a dual-space distillation framework that transfers procedural problem-solving ability from a large teacher model to compact student models. DuoMem distils in two complementary spaces: (1)context-space distillation, which replaces student-generated memories with higher-quality teacher-generated procedural memories prepended to the student's input, and (2)parameter-space distillation, which fine-tunes lightweight LoRA adapters on successful teacher trajectories. Evaluated on ALFWorld, a challenging embodied decision-making benchmark, DuoMem boosts a 4B-parameter model from 4.3% to 77.9% task success rate, closing most of the gap to a 72B teacher model (87.1%), while adding fewer than 10M trainable parameters and only a few megabytes of pre-computed teacher memories. Moreover, the DuoMem-enhanced 4B model completes tasks over 3x faster than the 72B teacher in wall-clock time, making it viable for real-time edge deployment, which would be challenging for the teacher.Extensive ablations across eight models spanning 2B-72B parameters reveal that both distillation axes contribute complementary

Related papers

• MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale [23.327794385289952]
  LLM agents execute heterogeneous sequences of model calls within a single task.<n>Prior routing methods assign easy invocations to a cheaper small model and difficult ones to a large model.<n> MERA instead improves the small model itself, using a single model invocation as the unit of adaptation.
  arXiv  Detail & Related papers  (2026-08-11T00:28:02Z)
• Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory [49.6908961629532]
  Agent Memory Distillation (AMD) is a training-free framework that transfers structured knowledge from a large teacher agent to a small student agent through hierarchical memory.<n>AMD constructs three complementary memory types from successful teacher trajectories: memory encodes task-level strategies, Subtask memory provides concrete behavioral examples, and Function memory captures per-function calling conventions.<n>We evaluate AMD on three tool-use benchmarks using four student models (4B-8B parameters) with GPT-5-mini as the teacher, achieving average accuracy gains of 27.2%p, 11.2%p, and 3.4%p on AppWorld, BFCL V
  arXiv  Detail & Related papers  (2026-08-07T12:43:00Z)
• AutoMem: Automated Learning of Memory as a Cognitive Skill [63.07128029793901]
  We show that memory expertise is a learned skill.<n>We introduce AutoMem, a framework that automates the two axes of memory management.<n>Our results show that memory management is an independently learnable skill.
  arXiv  Detail & Related papers  (2026-07-01T17:57:03Z)
• Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers [47.916191066149224]
  Bridge is a lightweight latent interface that maps Student latents into the Teacher space without modifying the Student backbone.<n>It improves SD 1.5 from 5.4 to 9.4 HPSv3 while preserving one-step inference, low latency, and broad ecosystem compatibility.
  arXiv  Detail & Related papers  (2026-06-30T17:51:26Z)
• Generalizable and Efficient Automated Scoring with a Knowledge-Distilled Multi-Task Mixture-of-Experts [5.109529226503146]
  UniMoE-Guided transfers expertise from multiple task-specific large models (teachers) into a single compact, deployable model (student)<n>The student combines (i) a shared encoder for cross-task representations, (ii) a gated MoE block that balances shared and task-specific processing, and (iii) lightweight task heads.
  arXiv  Detail & Related papers  (2025-11-18T04:55:44Z)
• Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning [89.55738101744657]
  Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless.<n>We present Memory-R1, a reinforcement learning framework that equips LLMs with the ability to actively manage and utilize external memory.
  arXiv  Detail & Related papers  (2025-08-27T12:26:55Z)
• UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning [22.029614513198663]
  Memory-layer architectures offer an appealing alternative with very few memory access.<n>We present UltraMemV2, a redesigned memory-layer architecture that closes this performance gap.<n>We demonstrate that UltraMemV2 performance parity with 8-expert MoE models under same computation and parameters but significantly low memory access.
  arXiv  Detail & Related papers  (2025-08-26T07:33:11Z)
• Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergence [18.640219880439062]
  This paper presents an innovative approach to leverage intermediate spatial representations.<n>We propose Asymmetric Decision-Making (ADM) to enhance feature consensus learning for student models.
  arXiv  Detail & Related papers  (2025-03-09T16:32:25Z)
• SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction [89.56181323849512]
  SuperCorrect is a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model.<n>In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts.<n>In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model.
  arXiv  Detail & Related papers  (2024-10-11T17:25:52Z)
• Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher [43.678380057638016]
  Gap Preserving Distillation (GPD) method trains an additional dynamic teacher model from scratch along with training the student to bridge this gap. In experiments, GPD significantly outperforms existing distillation methods on top of both CNNs and transformers architectures. GPD also generalizes well to the scenarios without a pre-trained teacher, including training from scratch and fine-tuning, yielding a large improvement of 1.80% and 0.89% on ResNet18.
  arXiv  Detail & Related papers  (2024-10-05T12:29:51Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning [146.51221523793342]
  LightPAFF uses two-stage knowledge distillation to transfer knowledge from a big teacher model to a lightweight student model. LightPAFF reduces the model size by nearly 5x and improves online inference speed by 5x-7x.
  arXiv  Detail & Related papers  (2020-04-27T14:00:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.