Spatio-Temporal Grounding of Large Language Models from Perception Streams

• URL: http://arxiv.org/abs/2604.07592v1
• Date: Wed, 08 Apr 2026 20:49:50 GMT
• Title: Spatio-Temporal Grounding of Large Language Models from Perception Streams
• Abstract summary: We introduce Formally Explainable Scenes (FESTS) that injects verifiable temporal supervision into Large Language Models.<n>Training a 3-billion-level model on 27k suchs boosts frame size from magnitude to 88.5%, matching GPT-4.1 on complex complex-temporal reasoning.
• Score: 3.426888015461207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied-AI agents must reason about how objects move and interact in 3-D space over time, yet existing smaller frontier Large Language Models (LLMs) still mis-handle fine-grained spatial relations, metric distances, and temporal orderings. We introduce the general framework Formally Explainable Spatio-Temporal Scenes (FESTS) that injects verifiable spatio-temporal supervision into an LLM by compiling natural-language queries into Spatial Regular Expression (SpRE) -- a language combining regular expression syntax with S4u spatial logic and extended here with universal and existential quantification. The pipeline matches each SpRE against any structured video log and exports aligned (query, frames, match, explanation) tuples, enabling unlimited training data without manual labels. Training a 3-billion-parameter model on 27k such tuples boosts frame-level F1 from 48.5% to 87.5%, matching GPT-4.1 on complex spatio-temporal reasoning while remaining two orders of magnitude smaller, and, hence, enabling spatio-temporal intelligence for Video LLM.

Related papers

• Conditional Multi-Event Temporal Grounding in Long-Form Video [19.941856770065854]
  Existing benchmarks fall short: they localize only a single moment per query, count without temporal conditions, or treat grounding and counting as disjoint tasks.<n>We introduce CoMET-Bench for Multi-Event Temporal Grounding in long-form video, comprising 2789 queries over 600 videos averaging 33.8 minutes across five real-world domains.<n>We propose CoMET-Agent, a training-free agentic framework that reformulates the task as structured search-and-aggregate, improving F1@0.5 by 6.1% over GPT-5 purely through structural reasoning.
  arXiv  Detail & Related papers  (2026-06-13T14:27:58Z)
• Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding [6.673833357112721]
  textbfBridge-STG is an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence.<n>Bridge-STG achieves state-of-the-art performance among MLLM-based methods.
  arXiv  Detail & Related papers  (2026-04-09T09:14:00Z)
• MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence [50.11889361459544]
  Humans are born with vision-based 4D spatial-temporal intelligence.<n>Despite its importance, this capability remains a significant bottleneck for current large language models (MLLMs)
  arXiv  Detail & Related papers  (2026-02-28T07:23:36Z)
• TOGGLE: Temporal Logic-Guided Large Language Model Compression for Edge [0.0]
  Large Language Models (LLMs) deliver exceptional performance across natural language tasks.<n>Existing compression techniques, such as quantization and pruning, often degrade critical linguistic properties.<n>We propose Temporal Logic-Guided Large Language Model Compression (TOGGLE) to formally specify and enforce linguistic properties during compression.
  arXiv  Detail & Related papers  (2025-12-18T18:27:42Z)
• NL2SpaTiaL: Generating Geometric Spatio-Temporal Logic Specifications from Natural Language for Manipulation Tasks [8.178210432886237]
  Spatio-Temporal Logic (SpaTiaL) offers a principled formalism for expressing geometric spatial requirements.<n>We introduce a dataset generation framework that synthesizes SpaTiaL specifications and converts them into natural-language descriptions.<n>This pipeline produces the NL2SpaTiaL dataset, aligning natural language with multi-level spatial relations and temporal objectives.
  arXiv  Detail & Related papers  (2025-12-15T18:56:34Z)
• Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data [100.5266292850922]
  Strefer is a synthetic data generation framework designed to equip Video Large Models with referring and reasoning capabilities.<n>Strefer produces diverse instruction-generation data using a data engine that pseudo-annotates temporally dense, fine-grained video metadata.<n>Our approach enhances the ability of Video LLMs to interpret to spatial and temporal references, fostering more versatile, space-time-aware reasoning essential for real-world AI companions.
  arXiv  Detail & Related papers  (2025-09-03T17:33:20Z)
• A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding [78.99798110890157]
  Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries.<n>Existing language field methods struggle to accurately localize instances using spatial relations in language queries.<n>We propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning.
  arXiv  Detail & Related papers  (2025-07-09T10:20:38Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability [58.46310813774538]
  Large language models (LMLMs) have made remarkable progress in either temporal or spatial localization.<n>However they struggle to perform-temporal video grounding.<n>This limitation stems from two major challenges.<n>We introduce SpaceLM, a MLLMVL endowed with temporal-temporal video grounding.
  arXiv  Detail & Related papers  (2025-03-18T07:40:36Z)
• LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding [29.42797944919497]
  We propose LLaVA-ST, a MLLM for fine-grained spatial-temporal multimodal understanding.<n>In LLaVA-ST, we propose Language-Aligned Positional Embedding, which embeds the coordinate special token into the visual space.<n>We also design the Spatial-Temporal Packer, which decouples the feature compression of temporal and spatial resolutions into two distinct point-to-region attention processing streams.
  arXiv  Detail & Related papers  (2025-01-14T17:58:12Z)
• Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model [51.83436609094658]
  We introduce Coarse Correspondences, a simple lightweight method that enhances MLLMs' spatial-temporal reasoning with 2D images as input. Our method uses a lightweight tracking model to identify primary object correspondences between frames in a video or across different image viewpoints. We demonstrate that this simple training-free approach brings substantial gains to GPT4-V/O consistently on four benchmarks.
  arXiv  Detail & Related papers  (2024-08-01T17:57:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.