SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards

• URL: http://arxiv.org/abs/2606.24726v1
• Date: Tue, 23 Jun 2026 15:50:33 GMT
• Title: SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards
• Abstract summary: We propose Semantic Evidence Reward (SER) to reform RL as a constrained task.<n>Instead of computing pixel-level annotations, SER uses a referee enhancing a VLM to evaluate model-generated evidence claims.<n>On the VLM benchmark, SER achieves 49.6%, improving by 3.0 points over the strong evidence-ground baseline.
• Score: 8.705689021755221
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video MLLMs often struggle with fine-grained spatio-temporal reasoning, sometimes generating correct answers based on irrelevant frames or objects. Although outputting spatio-temporal evidence during reasoning is a promising direction, existing RL frameworks typically rely on geometry-only (IoU) rewards, which can be sensitive to boundary perturbations and overlook semantic alignment. To address this, we propose Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. Instead of computing pixel-level overlap, SER uses a referee VLM as a local checker to evaluate model-generated evidence claims across two dimensions: relevance and localization quality, combined with a temporal penalty. This design reduces the reliance on dense box annotations and enables training directly on standard video QA data. On the V-STAR benchmark, SER achieves 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding.

Related papers

• Long-to-Short Video Evidence Reasoning for Grounded Question Answering [22.18808808973094]
  LOVER highlights three innovations over existing reinforcement-learning based video reasoning models.<n>Lover built on Time-R1 achieves new state-of-the-art results among open-source models on popular GQA benchmarks: NExT-GQA and ReXTime.
  arXiv  Detail & Related papers  (2026-09-14T08:43:27Z)
• Evidence-Backed Video Question Answering [90.93225758130426]
  Existing explainability efforts rely on textual rationales or sparse bounding boxes.<n>We propose Evidence-Backed Video Question Answering (E-VQA)<n>To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding.
  arXiv  Detail & Related papers  (2026-07-13T17:49:10Z)
• Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoning [37.09676301107883]
  We argue that spatial reasoning should be revisitable.<n>We propose ReRe, a training-free, inferencetime framework with two phases.<n>We show that ReRe substantially boosts open-source MLLMs to rival proprietary state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-06-10T05:52:14Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification [73.02304272829785]
  VideoBench is a hierarchical benchmark for long-video answering that rigorously verifies evidence.<n>It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence.<n>Experiments show that even GeminiPro-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting.<n>Results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-04-02T03:29:43Z)
• Reinforcing Consistency in Video MLLMs with Structured Rewards [14.560061824569333]
  Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding.<n>We study this failure mode through a compositional consistency audit that decomposes a caption into supporting factual and temporal claims.<n>Our training objective integrates three complementary components: (1) an instance-aware scene-graph reward for factual objects, attributes, and relations; (2) a temporal reward for event ordering and repetition; and (3) a video-grounded VQA reward for hierarchical self-verification.
  arXiv  Detail & Related papers  (2026-04-01T23:15:34Z)
• VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting [58.508924874097715]
  VisonCoach consists of two components: (1) Visual Prompt Selector, which predicts appropriate prompt types conditioned on the video and question, and (2) Spatio-Reasoner, optimized with RL under visual prompt guidance and object grounding rewards.<n>Our results show that visual prompting during training improves grounded video reasoning, while distract-distillationSense enables the model to this ability without requiring prompts at inference time.
  arXiv  Detail & Related papers  (2026-03-15T23:32:02Z)
• Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence [70.2803680525165]
  We introduce Open-o3 Video, a non-agent framework that integrates explicit evidence into video reasoning.<n>The model highlights key objects and bounding boxes alongside its answers, allowing reasoning to be grounded in concrete visual observations.<n>On V-STAR benchmark, Open-o3 Video achieves state-of-the-art performance, raising mAM by 14.4% and mL timestamp by 24.2%.
  arXiv  Detail & Related papers  (2025-10-23T14:05:56Z)
• When and What: Diffusion-Grounded VideoLLM with Entity Aware Segmentation for Long Video Understanding [12.410012029024342]
  We present Grounded VideoDiT, a Video LLM designed to overcome limitations by introducing three key innovations.<n>First, a Diffusion Temporal Latent (DTL) encoder enhances boundary sensitivity and maintains temporal consistency.<n>Second, object grounded representations explicitly bind query entities to localized visual evidence, strengthening alignment.<n>Third, a mixed token scheme with discrete temporal timestamp tokens provides explicit modeling, enabling fine grained temporal reasoning.
  arXiv  Detail & Related papers  (2025-08-21T15:12:14Z)
• Weakly-Supervised Video Object Grounding via Causal Intervention [82.68192973503119]
  We target at the task of weakly-supervised video object grounding (WSVOG), where only video-sentence annotations are available during model learning. It aims to localize objects described in the sentence to visual regions in the video, which is a fundamental capability needed in pattern analysis and machine learning.
  arXiv  Detail & Related papers  (2021-12-01T13:13:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.