Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)

• URL: http://arxiv.org/abs/2604.16395v2
• Date: Wed, 22 Apr 2026 19:11:26 GMT
• Title: Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)
• Abstract summary: Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency.<n>We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments.
• Score: 2.8350851063316873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Context retrieval systems for LLM inference face a critical challenge: high retrieval latency creates a fundamental tension between waiting for complete context (poor time-to-first-token) and proceeding without it (reduced quality). Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency, but doing so with concurrent requests introduces new challenges: requests contend for GPU compute and memory, and scheduling must adapt to dynamic context arrivals. We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments. Stream2LLM introduces adaptive scheduling and preemption for two distinct retrieval patterns: append-mode (progressive context accumulation) and update-mode (iterative refinement with cache invalidation). It decouples scheduling decisions from resource acquisition, enabling flexible preemption strategies guided by hardware-specific cost models, and uses longest common prefix matching to minimize redundant computation when input changes dynamically. To evaluate Stream2LLM, we collect two large-scale, real-world streaming workloads based on web crawling and approximate nearest neighbor search. Our evaluation demonstrates that streaming architecture delivers up to 11x TTFT improvements, with cost-aware scheduling providing critical benefits under memory pressure, all while maintaining throughput parity with non-streaming baselines. Code: https://github.com/rajveerb/stream2llm/tree/mlsys_artifact

Related papers

• ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding [16.068802214957014]
  Streaming video understanding is a critical capability for real-world applications.<n>ShallowStream is a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and retrieval index building.<n>ShallowStream achieves on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively.
  arXiv  Detail & Related papers  (2026-09-02T16:14:48Z)
• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation [63.50393485685279]
  We present a vLLM-based inference pipeline for unified speech understanding and generation.<n>We overcome the shared intuition-Free Guidance (CFG) halves throughput.<n>By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput.
  arXiv  Detail & Related papers  (2026-07-02T12:55:11Z)
• Efficient Learned Data Compression via Dual-Stream Feature Decoupling [11.752785837648199]
  We propose a Dual-Stream Multi-Scale Decoupler that disentangles local and global contexts to replace deep serial processing with shallow parallel streams.<n>Our method achieves state-of-the-art performance in both compression ratio and throughput, while maintaining the lowest latency and memory usage.
  arXiv  Detail & Related papers  (2026-04-08T16:05:24Z)
• Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models [14.21980212001207]
  Motivated by the streaming nature of video data, we investigate two streaming reasoning paradigms for LVLMs.<n>To better match streaming inputs, we propose textbfThink-as-You-See (TaYS), a unified framework enabling true concurrent reasoning.
  arXiv  Detail & Related papers  (2026-03-03T11:24:55Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• Accelerating Streaming Video Large Language Models via Hierarchical Token Compression [12.247532124314402]
  Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks.<n>They face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams.<n>We propose textbfStreaming textbfToken textbfCompression (textbfSTC), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs.
  arXiv  Detail & Related papers  (2025-11-30T13:44:28Z)
• StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA [60.86024022291499]
  We introduce StreamingCoT, the first dataset explicitly designed for temporally evolving reasoning in streaming video streams.<n>Our framework generates per-second dense descriptions and constructs temporally-dependent semantic segments through similarity fusion.<n>This dataset establishes a foundation for advancing research in streaming video understanding, complex temporal reasoning, and multimodal inference.
  arXiv  Detail & Related papers  (2025-10-29T09:47:38Z)
• Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling [57.708486655254966]
  Delayed Streams Modeling is a flexible formulation for sequence-to-sequence learning.<n>It provides streaming inference of arbitrary output sequences from any input combination.
  arXiv  Detail & Related papers  (2025-09-10T16:43:01Z)
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition [95.54406667705999]
  Pangu Embedded is an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs)<n>It addresses the significant computational costs and inference latency challenges prevalent in existing reasoning-optimized LLMs.<n>It delivers rapid responses and state-of-the-art reasoning quality within a single, unified model architecture.
  arXiv  Detail & Related papers  (2025-05-28T14:03:02Z)
• Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints [14.341123057506827]
  Large Language Models (LLMs) are indispensable in today's applications, but their inference procedure demands significant computational resources.<n>This paper formulates LLM inference optimization as a multi-stage online scheduling problem.<n>We develop a fluid dynamics approximation to provide a tractable benchmark that guides algorithm design.
  arXiv  Detail & Related papers  (2025-04-15T16:00:21Z)
• SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolution [10.896823447820232]
  We propose SSCFormer to push the limit of chunk-wise Conformer for streaming ASR. We show that an End-to-End (E2E) CER 5.33% can achieve, which even outperforms a strong time-restricted baseline U2.
  arXiv  Detail & Related papers  (2022-11-21T13:04:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.