Kinaema: a recurrent sequence model for memory and pose in motion

• URL: http://arxiv.org/abs/2510.20261v1
• Date: Thu, 23 Oct 2025 06:34:53 GMT
• Title: Kinaema: a recurrent sequence model for memory and pose in motion
• Abstract summary: We introduce a new model, Kinaema, and agent, capable of integrating a stream of visual observations while moving in a potentially large scene.<n>Our model does not explicitly store an observation history, therefore does not have hard constraints on context length.<n>We show that our large-capacity recurrent model maintains a useful representation of the scene, navigates to goals observed before the actual episode start, and is computationally efficient.
• Score: 25.677034515160567
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: One key aspect of spatially aware robots is the ability to "find their bearings", ie. to correctly situate themselves in previously seen spaces. In this work, we focus on this particular scenario of continuous robotics operations, where information observed before an actual episode start is exploited to optimize efficiency. We introduce a new model, Kinaema, and agent, capable of integrating a stream of visual observations while moving in a potentially large scene, and upon request, processing a query image and predicting the relative position of the shown space with respect to its current position. Our model does not explicitly store an observation history, therefore does not have hard constraints on context length. It maintains an implicit latent memory, which is updated by a transformer in a recurrent way, compressing the history of sensor readings into a compact representation. We evaluate the impact of this model in a new downstream task we call "Mem-Nav". We show that our large-capacity recurrent model maintains a useful representation of the scene, navigates to goals observed before the actual episode start, and is computationally efficient, in particular compared to classical transformers with attention over an observation history.

Related papers

• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• Compressing Observation History into Agent Memory: Distilling Transformers into Recurrent Transformers [25.583827940095322]
  We focus on long-horizon streaming vision and robotics applications like map-free pose estimation.<n>We argue that this gap does not stem from architectural limitations, but from differences in how these models learn to compress past information.<n>We show that this approach allows to train a recurrent latent robotic memory with linear-time complexity while substantially narrowing the performance gap to full-history transformers.
  arXiv  Detail & Related papers  (2026-06-19T15:58:36Z)
• GenFusion: Feed-forward Human Performance Capture via Progressive Canonical Space Updates [17.25826599799299]
  We present a feed-forward human performance capture method that renders novel views of a performer from a monocular RGB stream.<n>We take advantage of the fact that more body parts become observable by maintaining a canonical space that is progressively updated with each incoming frame.<n>This canonical space accumulates appearance information over time and serves as a context bank when direct observations are missing in the current live frame.
  arXiv  Detail & Related papers  (2026-03-30T20:55:00Z)
• Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction [60.964512894143475]
  We present Generative Spatial Transformer ( GST), a novel auto-regressive framework that jointly addresses spatial localization and view prediction. Our model simultaneously estimates the camera pose from a single image and predicts the view from a new camera pose, effectively bridging the gap between spatial awareness and visual prediction.
  arXiv  Detail & Related papers  (2024-10-24T17:58:05Z)
• A Robotics-Inspired Scanpath Model Reveals the Importance of Uncertainty and Semantic Object Cues for Gaze Guidance in Dynamic Scenes [8.64158103104882]
  We present a computational model that simulates object segmentation and gaze behavior in an interconnected manner.<n>We show how our model's modular design allows for extensions, such as incorporating saccadic momentum or pre-saccadic attention.
  arXiv  Detail & Related papers  (2024-08-02T15:20:34Z)
• Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories [28.701879490459675]
  We aim to learn an implicit motion field parameterized by a neural network to predict the movement of novel points within same domain. We exploit intrinsic regularization provided by SIREN, and modify the input layer to produce atemporally smooth motion field. Our experiments assess the model's performance in predicting unseen point trajectories and its application in temporal mesh alignment with deformation.
  arXiv  Detail & Related papers  (2024-06-05T21:02:10Z)
• HumMUSS: Human Motion Understanding using State Space Models [6.821961232645209]
  We propose a novel attention-free model for human motion understanding building upon recent advancements in state space models. Our model supports both offline and real-time applications. For real-time sequential prediction, our model is both memory efficient and several times faster than transformer-based approaches.
  arXiv  Detail & Related papers  (2024-04-16T19:59:21Z)
• Cached Transformers: Improving Transformers with Differentiable Memory Cache [71.28188777209034]
  This work introduces a new Transformer model called Cached Transformer. It uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens.
  arXiv  Detail & Related papers  (2023-12-20T03:30:51Z)
• Layout Sequence Prediction From Noisy Mobile Modality [53.49649231056857]
  Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. We propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories.
  arXiv  Detail & Related papers  (2023-10-09T20:32:49Z)
• Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving [68.95178518732965]
  A self-driving vehicle (SDV) must be able to perceive its surroundings and predict the future behavior of other traffic participants. Existing works either perform object detection followed by trajectory of the detected objects, or predict dense occupancy and flow grids for the whole scene. This motivates our unified approach to perception and future prediction that implicitly represents occupancy and flow over time with a single neural network.
  arXiv  Detail & Related papers  (2023-08-02T23:39:24Z)
• STAR: Sparse Transformer-based Action Recognition [61.490243467748314]
  This work proposes a novel skeleton-based human action recognition model with sparse attention on the spatial dimension and segmented linear attention on the temporal dimension of data. Experiments show that our model can achieve comparable performance while utilizing much less trainable parameters and achieve high speed in training and inference.
  arXiv  Detail & Related papers  (2021-07-15T02:53:11Z)
• History Repeats Itself: Human Motion Prediction via Motion Attention [81.94175022575966]
  We introduce an attention-based feed-forward network that explicitly leverages the observation that human motion tends to repeat itself. In particular, we propose to extract motion attention to capture the similarity between the current motion context and the historical motion sub-sequences. Our experiments on Human3.6M, AMASS and 3DPW evidence the benefits of our approach for both periodical and non-periodical actions.
  arXiv  Detail & Related papers  (2020-07-23T02:12:27Z)
• A Spatio-temporal Transformer for 3D Human Motion Prediction [39.31212055504893]
  We propose a Transformer-based architecture for the task of generative modelling of 3D human motion. We empirically show that this effectively learns the underlying motion dynamics and reduces error accumulation over time observed in auto-gressive models.
  arXiv  Detail & Related papers  (2020-04-18T19:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.