JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management

• URL: http://arxiv.org/abs/2609.17475v1
• Date: Tue, 15 Sep 2026 17:15:48 GMT
• Title: JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
• Abstract summary: We present an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTransSwap for state-preserving serving transitions.<n>The integrated runtime answers 29 of 30 AIME 2026 problems correctly.
• Score: 5.210656026160777
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Capable open-weight models make local coding and reasoning attractive, but their context and execution state strain laptop memory. We present JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions. These mechanisms fuse reconstruction and coordinate just-in-time materialization and release, independently of model-weight quantization. In full-execution capacity tests on a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, three independent runs complete 196,608 input and 16,384 output tokens, increasing completed single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x); a separate two-request run retains 229,376 positions in aggregate. In separate performance tests, a 32K-input, 64-output probe reaches 19.11 tokens/s, and a repeated 32K+6K workload has a median peak process footprint of 16,374 MiB. The integrated runtime answers 29 of 30 AIME 2026 problems correctly, showing how compact state and lifetime-aware execution expand local serving capacity while supporting extended generated reasoning.

Related papers

• FESC: Remodeling Long-Context Private Inference with Encrypted State-Space Models [21.63908755417832]
  We present Factorized Encrypted Scan-Contract (FESC), a hybrid FHE-MPC system for private long-context selective SSM inference.<n>FESC is the first private long-document inference system to complete native end-to-end execution at $L geq 1,024$ on a single GPU.
  arXiv  Detail & Related papers  (2026-08-18T07:20:09Z)
• QSimAdv: A Late-Bound, Vendor-Agnostic Architecture for High-Performance Quantum-Circuit Simulation [1.435000056979246]
  We present QSimAdv, which makes late binding, rather than a common kernel.<n>We realize this design on NVIDIA GH200 and AMD MI250X/EPYC systems.
  arXiv  Detail & Related papers  (2026-08-11T23:17:29Z)
• RoPE-Aware Bit Allocation for KV-Cache Quantization [52.099459337231345]
  Block-GTQ is a bit allocator for key-cache quantization built on TurboQuant-MSE.<n>It better preserves RoPE query-key logits on a ten-model diagnostic panel.<n>It runs 1.34x faster than fp16 FlashAttention2 at 128K context.
  arXiv  Detail & Related papers  (2026-06-23T00:17:48Z)
• MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research [39.68394140712043]
  MobileGym is a lightweight, fully controllable environment for everyday mobile use.<n>A layered state model and a declarative task-definition framework keep state programmability and task creation practical at scale.<n>A single programmatic judging mechanism delivers both deterministic evaluation verdicts and dense rewards.
  arXiv  Detail & Related papers  (2026-05-25T17:59:49Z)
• quantum-safe: Bridging the Post-Quantum Production Gap with a Hybrid-by-Default Python Cryptography Library [0.0]
  quantum-safe is a Python that closes all three critical gaps in post-quantum cryptography.<n>We score nine PQC libraries across eight production-readiness dimensions.<n>A full X25519 + ML-KEM-768 handshake completes in 243 s under Docker/ Linux.
  arXiv  Detail & Related papers  (2026-05-16T16:05:11Z)
• Measuring Maximum Activations in Open Large Language Models [60.3514350516308]
  We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
  arXiv  Detail & Related papers  (2026-05-15T03:31:51Z)
• Native LLM and MLLM Inference at Scale on Apple Silicon [0.8122270502556375]
  We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built native on MLX.<n>For text models, we achieve 21% to 87% higher throughput than llama- across models ranging from Qwen3-0.6B to Nemotron-30B.<n>For multimodal models, we introduce content-based caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format.
  arXiv  Detail & Related papers  (2026-01-27T03:11:02Z)
• dParallel: Learnable Parallel Decoding for dLLMs [77.24184219948337]
  Diffusion large language models (dLLMs) offer parallel token prediction and lower inference latency.<n>Existing open-source models still require nearly token-length decoding steps to ensure performance.<n>We introduce dParallel, a simple and effective method that unlocks the inherent parallelism of dLLMs for fast sampling.
  arXiv  Detail & Related papers  (2025-09-30T16:32:52Z)
• ParallelComp: Parallel Long-Context Compressor for Length Extrapolation [51.68913021512016]
  Extrapolating ultra-long contexts (text length >128K) remains a major challenge for large language models (LLMs)<n>In this work, we propose ParallelComp, a parallel long-context compression method that effectively overcomes the memory bottleneck.<n>We achieve a 1.76x improvement in chunk throughput, thereby achieving a 23.50x acceleration in the prefill stage with negligible performance loss.
  arXiv  Detail & Related papers  (2025-02-20T07:10:43Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• LoCoCo: Dropping In Convolutions for Long Context Compression [77.26610232994508]
  This paper presents a novel approach, Dropping In Convolutions for Long Context Compression (LoCoCo) LoCoCo employs only a fixed-size Key-Value ( KV) cache, and can enhance efficiency in both inference and fine-tuning stages.
  arXiv  Detail & Related papers  (2024-06-08T01:35:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.