H$^{2}$MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer

• URL: http://arxiv.org/abs/2605.24930v1
• Date: Sun, 24 May 2026 08:22:05 GMT
• Title: H$^{2}$MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer
• Abstract summary: H2MT makes long-context inference structure-aware.<n>It builds a semantic hierarchy offline, computes a memory embedding for each node via bottom-up post-order aggregation.<n>It delivers competitive ROUGE-L and F1 with lower peak GPU memory and time-to-first-token (TTFT) than prompt compression, memory-token methods, and retrieval-augmented generation baselines.
• Score: 42.43543809277424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based LLMs achieve strong results on many language tasks; however, long inputs remain challenging because context windows are finite, and prefill latency and memory grow rapidly with prompt length. Flat token-stream processing and chunk-based retrieval can therefore spend substantial computation and context budget on text unrelated to the query. Offline-indexed RAG additionally introduces external storage and index management overhead, and typically appends retrieved evidence as raw text, increasing prefill cost and latency. H^{2}MT makes long-context inference structure-aware: it builds a semantic hierarchy offline, computes a memory embedding for each node via bottom-up post-order aggregation, and routes queries coarse-to-fine at inference to prune irrelevant branches early. On LongBench QA (NarrativeQA, HotpotQA, QASPER) and two structured technical-document settings, H MT achieves favorable quality efficiency trade-offs, delivering competitive ROUGE-L and F1 (where applicable) with lower peak GPU memory and time-to-first-token (TTFT) than prompt compression, memory-token methods, and retrieval-augmented generation baselines.

Related papers

• HingeMem: Boundary Guided Long-Term Memory with Query Adaptive Retrieval for Scalable Dialogues [12.529136884264092]
  HingeMem is a boundary-guided long-term memory that builds an interpretable indexing interface via boundary-triggered hyperedges.<n> query-adaptive retrieval mechanisms jointly decide (a) textitwhat to retrieve: determine the query-conditioned routing over the element-indexed memory.<n>We show that HingeMem achieves approximately $20%$ relative improvement over strong baselines without query categories specification.
  arXiv  Detail & Related papers  (2026-04-08T09:07:07Z)
• Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT) [2.8350851063316873]
  Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency.<n>We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments.
  arXiv  Detail & Related papers  (2026-03-29T06:49:12Z)
• Panini: Continual Learning in Token Space via Structured Memory [4.979820180013486]
  Language models are increasingly used to reason over content they were not trained on.<n>A common approach is retrieval-augmented generation (RAG), which stores verbatim documents externally (as chunks) and retrieves only a relevant subset at inference time.<n>We propose a human-like non-parametric continual learning framework, where the base model remains fixed, and learning occurs by integrating each new experience into an external semantic memory state.
  arXiv  Detail & Related papers  (2026-02-16T19:58:03Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding [55.45331924836242]
  We present URaG, a framework that Unifies Retrieval and Generation within a single MLLM.<n>We show that URaG achieves state-of-the-art performance while reducing computational overhead by 44-56%.
  arXiv  Detail & Related papers  (2025-11-13T17:54:09Z)
• Towards Hyper-Efficient RAG Systems in VecDBs: Distributed Parallel Multi-Resolution Vector Search [5.216774377033164]
  We propose textbfSemantic Pyramid Indexing (SPI), a novel multi-resolution vector indexing framework for RAG in VecDBs.<n>Unlike existing hierarchical methods that require offline tuning or separate model training, SPI constructs a semantic pyramid over document embeddings and dynamically selects the optimal resolution level per query.<n>We implement SPI as a plugin for both FAISS and Qdrant backends and evaluate it across multiple RAG tasks including MS MARCO, Natural Questions, and multimodal retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-11-12T09:31:08Z)
• MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning [73.27233666920618]
  We propose MemSearcher, an agent workflow that iteratively maintains a compact memory and combines the current turn with it.<n>At each turn, MemSearcher fuses the user's question with the memory to generate reasoning traces, perform search actions, and update memory to retain only information essential for solving the task.<n>We introduce multi-context GRPO, an end-to-end RL framework that jointly optimize reasoning, search strategies, and memory management of MemSearcher Agents.
  arXiv  Detail & Related papers  (2025-11-04T18:27:39Z)
• ELITE: Embedding-Less retrieval with Iterative Text Exploration [5.8851517822935335]
  Large Language Models (LLMs) have achieved impressive progress in natural language processing.<n>Their limited ability to retain long-term context constrains performance on document-level or multi-turn tasks.
  arXiv  Detail & Related papers  (2025-05-17T08:48:43Z)
• MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation [60.04380907045708]
  Retrieval-Augmented Generation (RAG) is considered a promising strategy to address this problem.<n>We propose MemoRAG, a novel RAG framework empowered by global memory-augmented retrieval.<n>MemoRAG achieves superior performances across a variety of long-context evaluation tasks.
  arXiv  Detail & Related papers  (2024-09-09T13:20:31Z)
• Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs [61.40047491337793]
  We present Hierarchical cOntext MERging (HOMER), a new training-free scheme designed to overcome the limitations of large language models. HomeR uses a divide-and-conquer algorithm, dividing long inputs into manageable chunks. A token reduction technique precedes each merging, ensuring memory usage efficiency.
  arXiv  Detail & Related papers  (2024-04-16T06:34:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.