Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum

• URL: http://arxiv.org/abs/2608.30584v1
• Date: Mon, 31 Aug 2026 11:01:31 GMT
• Title: Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum
• Abstract summary: Existing evaluations and training data focus on simple queries.<n> compositional queries prevalent in real-world scenarios, where a target must be disambiguated by its attributes and relations to other entities.<n>We build a synthetic data engine that leverages a-temporal scene graph as a difficulty measure and casts difficulty-controlled queryBench as a constraint programming problem.<n>Built on this engine, we introduce ST-CompBench, a benchmark by explicit difficulty levels that capture temporal complexity and spatial interference.
• Score: 19.13127229043393
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Despite the impressive progress of recent MLLMs on spatio-temporal video grounding (STVG), existing evaluations and training data focus primarily on simple queries. They largely overlook the compositional queries prevalent in real-world scenarios, where a target must be disambiguated by jointly reasoning about its attributes and relations to other entities. To bridge this gap, we propose Compositional Spatio-Temporal Video Grounding (CompSTVG), a task that requires models to process complex textual queries where every intertwined attribute and relational cue is essential for disambiguation. To facilitate this task at scale, we build a synthetic data engine that leverages a spatio-temporal scene graph as a difficulty measure and casts difficulty-controlled query synthesis as a constraint programming problem, producing difficulty-graded data for both evaluation and training. Built on this engine, we introduce STVG-CompBench, a benchmark stratified by explicit difficulty levels that jointly capture temporal complexity and spatial interference. Evaluating 11 representative STVG models on STVG-CompBench reveals that current models perform poorly on compositional queries, exhibiting a sharp performance drop that is typically obscured by overall dataset-level averages. We further construct synthetic training data and propose CurrSTVG, a curriculum reinforcement learning framework that delivers consistent gains, with the largest improvements observed on the most challenging compositional queries.

Related papers

• LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks [58.3639630490749]
  We introduce LiveClawBench, a benchmark to evaluate LLM agents on real-world assistant tasks.<n>Based on an analysis of various realClaw usage cases, we derive a Triple-Axis Complexity Framework.<n>We construct a pilot benchmark with explicit complexity-factor annotations, covering real-world assistant tasks.
  arXiv  Detail & Related papers  (2026-03-20T16:08:21Z)
• Learning to Solve Complex Problems via Dataset Decomposition [53.1641602054716]
  This research explores a reverse curriculum generation approach that decomposes complex datasets into simpler, more learnable components.<n>We propose a teacher-student framework where the teacher is equipped with the ability to reason step-by-step, which is used to generate easier versions of examples.
  arXiv  Detail & Related papers  (2026-02-23T19:25:40Z)
• LLM-Symbolic Integration for Robust Temporal Tabular Reasoning [69.27153114778748]
  We introduce TempTabQA-C, a synthetic dataset designed for systematic and controlled evaluations.<n>This structured approach allows Large Language Models (LLMs) to generate and executesql queries, enhancing generalization and mitigating biases.
  arXiv  Detail & Related papers  (2025-06-06T05:14:04Z)
• SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields [11.872753517172555]
  We present SCENT, a novel framework for scalable and continuity-informed modeling learning.<n>SCENT unifies representation, reconstruction, and forecasting within a single architecture.<n>We validate SCENT through extensive simulations and real-world experiments, demonstrating state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-04-16T17:17:31Z)
• Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models [104.17057231661371]
  Time series analysis is crucial for understanding dynamics of complex systems.<n>Recent advances in foundation models have led to task-agnostic Time Series Foundation Models (TSFMs) and Large Language Model-based Time Series Models (TSLLMs)<n>Their success depends on large, diverse, and high-quality datasets, which are challenging to build due to regulatory, diversity, quality, and quantity constraints.<n>This survey provides a comprehensive review of synthetic data for TSFMs and TSLLMs, analyzing data generation strategies, their role in model pretraining, fine-tuning, and evaluation, and identifying future research directions.
  arXiv  Detail & Related papers  (2025-03-14T13:53:46Z)
• Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding [24.650102499933514]
  We focus on Weakly Supervised S-Temporal Video Grounding (WSTVG)<n>We first explore the potential of state-of-the-art object detection models for WSTVG.<n>Despite their robust zero-shot capabilities, our adaptation reveals significant limitations.<n>We propose CoSPaL, a novel approach which is designed to overcome these limitations.
  arXiv  Detail & Related papers  (2025-01-28T16:25:10Z)
• Understanding Synthetic Context Extension via Retrieval Heads [51.8869530817334]
  We investigate fine-tuning on synthetic data for three long-context tasks that require retrieval and reasoning.<n>We find that models trained on synthetic data fall short of the real data, but surprisingly, the mismatch can be interpreted.<n>Our results shed light on how to interpret synthetic data fine-tuning performance and how to approach creating better data for learning real-world capabilities over long contexts.
  arXiv  Detail & Related papers  (2024-10-29T17:55:00Z)
• RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs [12.846097618151951]
  We develop a dataset for LLMs Complex Reasoning over Textual Knowledge Graphs (RiTeK) with a broad topological structure coverage. We synthesize realistic user queries that integrate diverse topological structures, annotated information, and complex textual descriptions. We introduce an enhanced Monte Carlo Tree Search (CTS) method, which automatically extracts relational path information from textual graphs for specific queries.
  arXiv  Detail & Related papers  (2024-10-17T19:33:37Z)
• EasyST: A Simple Framework for Spatio-Temporal Prediction [18.291117879544945]
  We propose a simple framework for spatial-temporal prediction - EasyST paradigm. It learns lightweight and robust Multi-Layer Perceptrons (MLPs) generalization by distilling knowledge from complex-temporal GNNs. EasyST surpasses state-of-the-art approaches in terms of efficiency and accuracy.
  arXiv  Detail & Related papers  (2024-09-10T11:40:01Z)
• NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities [51.07379913779232]
  NeedleBench is a framework for assessing retrieval and reasoning performance in long-context tasks.<n>It embeds key data points at varying depths to rigorously test model capabilities.<n>Our experiments reveal that reasoning models like Deep-R1 and OpenAI's o3 struggle with continuous retrieval and reasoning in information-dense scenarios.
  arXiv  Detail & Related papers  (2024-07-16T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.