RT-Cache: Training-Free Retrieval for Real-Time Manipulation

• URL: http://arxiv.org/abs/2505.09040v3
• Date: Mon, 25 Aug 2025 00:15:27 GMT
• Title: RT-Cache: Training-Free Retrieval for Real-Time Manipulation
• Abstract summary: RT-Cache is a training-free retrieval-as-control pipeline for robots.<n>It embeds the current frame to retrieve and replay multi-step snippets, replacing per-step model calls.<n>A hierarchical search keeps lookups sub-second at million scale, shifting cost from compute to storage.
• Score: 13.338672711391999
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real robots are expected to repeat the same behavior in new environments with very little new data, yet modern controllers either incur heavy per-step inference or require deployment-time fine-tuning. We propose RT-Cache, a training-free retrieval-as-control pipeline that caches diverse image action trajectories in a unified vector memory and, at test time, embeds the current frame to retrieve and replay multi-step snippets, replacing per-step model calls. A hierarchical search keeps lookups sub-second at million scale, shifting cost from compute to storage and enabling real-time control on modest GPUs. Across real-robot tasks and large open logs, RT-Cache achieves higher success and lower completion time than strong retrieval baselines (approximately x2 higher success and ~30% faster in our settings), and a single-episode anchoring study shows immediate adaptation to a more complex, contact-rich task without fine-tuning. RT-Cache turns experience into an append-only memory, offering a simple, scalable path to few-shot deployment today and a foundation for multimodal keys and optional integration with high-level policies. Project page: https://rt-cache.github.io/.

Related papers

• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Scaling Short-Term Memory of Visuomotor Policies for Long-Horizon Tasks [69.19366746169906]
  We present PRISM, a transformer-based architecture for visuomotor policies to use short-term memory.<n>Piracy filters retrieved information to suppress irrelevant details, improving performance.<n> hierarchical architecture compresses local information into compact tokens, improving its compute and memory footprint.
  arXiv  Detail & Related papers  (2026-06-15T03:49:23Z)
• Transform-Invariant Generative Ray Path Sampling for Efficient Radio Propagation Modeling [5.567300300236187]
  Ray tracing has become a standard for accurate radio propagation modeling, but suffers from computational complexity.<n>We propose a comprehensive machine-learning-assisted framework that replaces exhaustive path searching with intelligent sampling via Generative Flow Networks.
  arXiv  Detail & Related papers  (2026-03-02T09:37:34Z)
• AMA: Adaptive Memory via Multi-Agent Collaboration [54.490349689939166]
  We propose Adaptive Memory via Multi-Agent Collaboration (AMA), a novel framework that leverages coordinated agents to manage memory across multiple granularities.<n>AMA significantly outperforms state-of-the-art baselines while reducing token consumption by approximately 80% compared to full-context methods.
  arXiv  Detail & Related papers  (2026-01-28T08:09:49Z)
• Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks [1.2292307778008844]
  We present a comprehensive evaluation of prompt caching across three major Large Language Model (LLM) providers.<n>Our results demonstrate that prompt caching reduces API costs by 45-80% and improves time to first token by 13-31% across providers.
  arXiv  Detail & Related papers  (2026-01-09T18:41:57Z)
• TRACER: Efficient Object Re-Identification in Networked Cameras through Adaptive Query Processing [8.955401552705892]
  Spatula is the state-of-the-art video database management system (VDBMS) for processing Re-ID queries.<n>It is not suitable for critical video analytics applications that require high recall due to camera history.<n>We present Tracer, a novel VDBMS for efficiently processing Re-ID queries using an adaptive query processing framework.
  arXiv  Detail & Related papers  (2025-07-13T02:22:08Z)
• VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers [23.868483243482558]
  We introduce an innovative vectorization based action tokenizer, leveraging over 100 times more data than previous approaches.<n>Once trained, the tokenizer can be seamlessly adapted to a wide range of tasks.<n>We conducted extensive experiments in both simulated environments and on real robotic platforms.
  arXiv  Detail & Related papers  (2025-07-01T17:59:44Z)
• FindingDory: A Benchmark to Evaluate Memory in Embodied Agents [49.89792845476579]
  We introduce a new benchmark for long-range embodied tasks in the Habitat simulator.<n>This benchmark evaluates memory-based capabilities across 60 tasks requiring sustained engagement and contextual awareness.
  arXiv  Detail & Related papers  (2025-06-18T17:06:28Z)
• PentaRAG: Large-Scale Intelligent Knowledge Retrieval for Enterprise LLM Applications [5.4838799162708245]
  We present PentaRAG, a five-layer module that routes each query through two instant caches.<n>We show that PentaRAG cuts average GPU time to 0.248 seconds per query, roughly half that of a naive RAG baseline.<n>Results demonstrate that a layered routing strategy can deliver freshness, speed, and efficiency simultaneously in production-grade RAG systems.
  arXiv  Detail & Related papers  (2025-06-18T07:54:53Z)
• Sparse Convolutional Recurrent Learning for Efficient Event-based Neuromorphic Object Detection [4.362139927929203]
  We propose the Sparse Event-based Efficient Detector (SEED) for efficient event-based object detection on neuromorphic processors.<n>We introduce sparse convolutional recurrent learning, which achieves over 92% activation sparsity in recurrent processing, vastly reducing the cost for reasoning on sparse event data.
  arXiv  Detail & Related papers  (2025-06-16T12:54:27Z)
• QuantCache: Adaptive Importance-Guided Quantization with Hierarchical Latent and Layer Caching for Video Generation [84.91431271257437]
  Diffusion Transformers (DiTs) have emerged as a dominant architecture in video generation.<n>DiTs come with significant drawbacks, including increased computational and memory costs.<n>We propose QuantCache, a novel training-free inference acceleration framework.
  arXiv  Detail & Related papers  (2025-03-09T10:31:51Z)
• Synthetica: Large Scale Synthetic Data for Robot Perception [21.415878105900187]
  We present Synthetica, a method for large-scale synthetic data generation for training robust state estimators. This paper focuses on the task of object detection, an important problem which can serve as the front-end for most state estimation problems. We leverage data from a ray-tracing, generating 2.7 million images, to train highly accurate real-time detection transformers. We demonstrate state-of-the-art performance on the task of object detection while having detectors that run at 50-100Hz which is 9 times faster than the prior SOTA.
  arXiv  Detail & Related papers  (2024-10-28T15:50:56Z)
• Why Sample Space Matters: Keyframe Sampling Optimization for LiDAR-based Place Recognition [6.468510459310326]
  We introduce the concept of sample space and propose a novel sampling approach for LiDAR-based place recognition.<n>Our approach demonstrates robust performance across diverse datasets, with the ability to adapt seamlessly from indoor to outdoor scenarios.
  arXiv  Detail & Related papers  (2024-10-03T16:29:47Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)
• RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation [11.321659218769598]
  Retrieval-Augmented Generation (RAG) has shown significant improvements in various natural language processing tasks. RAGCache organizes the intermediate states of retrieved knowledge in a knowledge tree and caches them in the GPU and host memory hierarchy. RAGCache reduces the time to first token (TTTF) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.
  arXiv  Detail & Related papers  (2024-04-18T18:32:30Z)
• Exploring Dynamic Transformer for Efficient Object Tracking [58.120191254379854]
  We propose DyTrack, a dynamic transformer framework for efficient tracking.<n>DyTrack automatically learns to configure proper reasoning routes for various inputs, gaining better utilization of the available computational budget.<n>Experiments on multiple benchmarks demonstrate that DyTrack achieves promising speed-precision trade-offs with only a single model.
  arXiv  Detail & Related papers  (2024-03-26T12:31:58Z)
• REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation [61.7171775202833]
  We introduce an efficient system for learning dexterous manipulation skills withReinforcement learning. The main idea of our approach is the integration of recent advances in sample-efficient RL and replay buffer bootstrapping. Our system completes the real-world training cycle by incorporating learned resets via an imitation-based pickup policy.
  arXiv  Detail & Related papers  (2023-09-06T19:05:31Z)
• R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics [9.2327813168753]
  This paper presents R3, a holistic solution for managing timing, memory, and algorithm performance in on-device real-time DRL training. R3 employs (i) a deadline-driven feedback loop with dynamic batch sizing for optimizing timing, (ii) efficient memory management to reduce memory footprint and allow larger replay buffer sizes, and (iii) a runtime coordinator guided by runtime analysis and a runtime profiler for adjusting memory resource reservations.
  arXiv  Detail & Related papers  (2023-08-29T05:48:28Z)
• $\Delta$-Patching: A Framework for Rapid Adaptation of Pre-trained Convolutional Networks without Base Performance Loss [71.46601663956521]
  Models pre-trained on large-scale datasets are often fine-tuned to support newer tasks and datasets that arrive over time. We propose $Delta$-Patching for fine-tuning neural network models in an efficient manner, without the need to store model copies. Our experiments show that $Delta$-Networks outperform earlier model patching work while only requiring a fraction of parameters to be trained.
  arXiv  Detail & Related papers  (2023-03-26T16:39:44Z)
• Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials [97.95400776235736]
  We present a framework based on offline RL that attempts to effectively learn new tasks. It combines pre-training on existing robotic datasets with rapid fine-tuning on a new task, with as few as 10 demonstrations. To our knowledge, PTR is the first RL method that succeeds at learning new tasks in a new domain on a real WidowX robot.
  arXiv  Detail & Related papers  (2022-10-11T06:30:53Z)
• A Fast Post-Training Pruning Framework for Transformers [74.59556951906468]
  Pruning is an effective way to reduce the huge inference cost of large Transformer models. Prior work on model pruning requires retraining the model. We propose a fast post-training pruning framework for Transformers that does not require any retraining.
  arXiv  Detail & Related papers  (2022-03-29T07:41:11Z)
• Virtual Replay Cache [20.531576904743282]
  We propose a new data structure, the Virtual Replay Cache (VRC), to address these shortcomings. VRC nearly eliminates DQN(lambda)'s cache memory footprint and slightly reduces the total training time on our hardware.
  arXiv  Detail & Related papers  (2021-12-06T23:40:27Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• A Framework for Efficient Robotic Manipulation [79.10407063260473]
  We show that a single robotic arm can learn sparse-reward manipulation policies from pixels. We show that, given only 10 demonstrations, a single robotic arm can learn sparse-reward manipulation policies from pixels.
  arXiv  Detail & Related papers  (2020-12-14T22:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.