Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models

• URL: http://arxiv.org/abs/2605.08974v1
• Date: Sat, 09 May 2026 14:32:36 GMT
• Title: Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models
• Abstract summary: Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
• Score: 154.39583176906893
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We argue this stems from a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors. To rigorously diagnose this, we introduce STEMO-Bench (Spatio-TEmporal MOnitoring), a benchmark of human-verified object-centric facts that evaluates intermediate reasoning by decomposing queries into sub-questions, distinguishing genuine temporal understanding from coincidental correctness. To address failure modes exposed by STEMO, we propose STEMO-Track, a novel object-centric framework that explicitly constructs and reasons over structured object trajectories via chunk-wise state extraction and temporal aggregation. Extensive experiments demonstrate that our object-centric framework significantly reduces hallucinated answers and improves spatio-temporal reasoning consistency over state-of-the-art MLLMs.

Related papers

• What's the Catch? Evaluating Temporal Consistency in Vision-Language Models [0.11121828451787184]
  Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal structure.<n>We introduce TimeCatch, where temporal anomalies are created by swapping consecutive frames and frame-level anomalies by replacing a frame with Gaussian noise.<n>Our evaluation reveals a substantial gap between frame-level and temporal anomaly detection.
  arXiv  Detail & Related papers  (2026-08-24T16:40:49Z)
• Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning [28.325120357282156]
  Video temporal-logical reasoning requires models to maintain, update, and compose evidence as visual states evolve across frames.<n>We introduce Video-MME-Logical, a benchmark organized around five temporal-logical operations.<n>It enables difficulty-controlled final-answer evaluation by varying temporal horizon and reasoning complexity.
  arXiv  Detail & Related papers  (2026-06-26T08:12:04Z)
• LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection [7.614138458711612]
  Vision-language models (VLMs) have emerged as a promising paradigm for video anomaly detection (VAD)<n>In this paper, we aim to address a key limitation of such pipelines, which perform segment-level inference independently owing to token constraints and reason without structured temporal context.<n>We propose a context-aware framework named LATERN, which reformulates VAD as a temporal evidence aggregation process.
  arXiv  Detail & Related papers  (2026-05-14T16:48:03Z)
• TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models [10.22422531424139]
  Video large language models (Video-LLMs) have achieved remarkable progress in general video understanding.<n>We introduce a diagnostic benchmark specifically designed to evaluate temporal object consistency in Video-LLMs.<n>To ensure that benchmark items depend on temporally ordered visual evidence, we propose a three-layer temporal-necessity filtering protocol.
  arXiv  Detail & Related papers  (2026-05-11T02:47:59Z)
• A Progressive Training Strategy for Vision-Language Models to Counteract Spatio-Temporal Hallucinations in Embodied Reasoning [49.61652671596548]
  "Multi-image hallucination reasoning", where a massive performance drop between forward and temporal queries reveals a dependence on superficial shortcuts instead of genuine understanding.<n>To mitigate this, we develop a new dataset: Chain-of-temporal building on detailed reasoning into Chain steps and definitive judgments.<n>Our experiments demonstrate that this approach not only improves accuracy but also the forward-backward performance gap from over 70% to only 6.53%.
  arXiv  Detail & Related papers  (2026-04-12T07:48:44Z)
• LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection [11.420370070522205]
  We introduce LiveFact, a benchmark that simulates the real-world "fog of war" in misinformation detection.<n>LiveFact uses dynamic, temporal evidence sets to evaluate models on their ability to reason with evolving, incomplete information.<n>Tests show that open-source Mixture-of-Experts models, such as Qwen3-235B-A22B, now match or outperform proprietary state-of-the-art systems.
  arXiv  Detail & Related papers  (2026-04-06T16:20:47Z)
• Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence [22.019359275289485]
  It is unclear whether large language models can perform abstracttemporal reasoning.<n>We present VA-BENCH, a benchmark comprising abstractive reasoning tasks together with their extractive counterparts.
  arXiv  Detail & Related papers  (2026-03-13T15:40:42Z)
• Multimodal Fact-Level Attribution for Verifiable Reasoning [80.60864342985748]
  Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation.<n>Existing multimodal grounding benchmarks and evaluation methods fail to assess attribution in complex multimodal reasoning.<n>We introduce MuRGAt, a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation.
  arXiv  Detail & Related papers  (2026-02-12T03:10:02Z)
• NarrativeTrack: Evaluating Video Language Models Beyond the Frame [10.244330591706744]
  We introduce NarrativeTrack, the first benchmark to evaluate narrative understanding in MLLMs.<n>We decompose videos into constituent entities and examine their continuity via a Compositional Reasoning (CRP) framework.<n>CRP challenges models to advance from temporal persistence to contextual evolution and fine-grained perceptual reasoning.
  arXiv  Detail & Related papers  (2026-01-03T07:12:55Z)
• Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding [56.7383554589569]
  Long video understanding is essential for human-like intelligence, enabling coherent perception and reasoning over extended temporal contexts.<n>We propose SpecTemp, a reinforcement learning-based Speculative Temporal reasoning framework.<n>We show that SpecTemp not only maintains competitive accuracy but also significantly accelerates inference compared with existing thinking-with-frames methods.
  arXiv  Detail & Related papers  (2025-11-30T09:27:59Z)
• Context-aware TFL: A Universal Context-aware Contrastive Learning Framework for Temporal Forgery Localization [60.73623588349311]
  We propose a universal context-aware contrastive learning framework (UniCaCLF) for temporal forgery localization.<n>Our approach leverages supervised contrastive learning to discover and identify forged instants by means of anomaly detection.<n>An efficient context-aware contrastive coding is introduced to further push the limit of instant feature distinguishability between genuine and forged instants.
  arXiv  Detail & Related papers  (2025-06-10T06:40:43Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• Learning to Track with Object Permanence [61.36492084090744]
  We introduce an end-to-end trainable approach for joint object detection and tracking. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI, and MOT17 datasets.
  arXiv  Detail & Related papers  (2021-03-26T04:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.