TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision

• URL: http://arxiv.org/abs/2506.09445v1
• Date: Wed, 11 Jun 2025 06:52:31 GMT
• Title: TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision
• Authors: Ayush Gupta, Anirban Roy, Rama Chellappa, Nathaniel D. Bastian, Alvaro Velasquez, Susmit Jha,
• Abstract summary: We address the problem of video question answering (video QA) with temporal grounding in a weakly supervised setup.<n>We generate an open-ended answer grounded with the start and end time.<n>We instruct-tune TOGA to jointly generate the answer and the temporal grounding.
• Score: 47.12557166147296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the problem of video question answering (video QA) with temporal grounding in a weakly supervised setup, without any temporal annotations. Given a video and a question, we generate an open-ended answer grounded with the start and end time. For this task, we propose TOGA: a vision-language model for Temporally Grounded Open-Ended Video QA with Weak Supervision. We instruct-tune TOGA to jointly generate the answer and the temporal grounding. We operate in a weakly supervised setup where the temporal grounding annotations are not available. We generate pseudo labels for temporal grounding and ensure the validity of these labels by imposing a consistency constraint between the question of a grounding response and the response generated by a question referring to the same temporal segment. We notice that jointly generating the answers with the grounding improves performance on question answering as well as grounding. We evaluate TOGA on grounded QA and open-ended QA tasks. For grounded QA, we consider the NExT-GQA benchmark which is designed to evaluate weakly supervised grounded question answering. For open-ended QA, we consider the MSVD-QA and ActivityNet-QA benchmarks. We achieve state-of-the-art performance for both tasks on these benchmarks.

Related papers

• MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering [64.46361702927457]
  Grounded Video Question Answering (Grounded VideoQA) requires aligning textual answers with explicit visual evidence.<n>In this work, we propose MUPA, a cooperative MUlti-Path Agentic approach that unifies video grounding, question answering, answer reflection and aggregation.
  arXiv  Detail & Related papers  (2025-06-22T15:39:02Z)
• Can I Trust Your Answer? Visually Grounded Video Question Answering [88.11169242115416]
  We study visually grounded VideoQA in response to the emerging trends of utilizing pretraining techniques for video-language understanding. We construct NExT-GQA -- an extension of NExT-QA with 10.5$K$ temporal grounding labels tied to the original QA pairs.
  arXiv  Detail & Related papers  (2023-09-04T03:06:04Z)
• Locate before Answering: Answer Guided Question Localization for Video Question Answering [70.38700123685143]
  LocAns integrates a question locator and an answer predictor into an end-to-end model. It achieves state-of-the-art performance on two modern long-term VideoQA datasets.
  arXiv  Detail & Related papers  (2022-10-05T08:19:16Z)
• Frame-Subtitle Self-Supervision for Multi-Modal Video Question Answering [73.11017833431313]
  Multi-modal video question answering aims to predict correct answer and localize the temporal boundary relevant to the question. We devise a weakly supervised question grounding (WSQG) setting, where only QA annotations are used. We transform the correspondence between frames and subtitles to Frame-Subtitle (FS) self-supervision, which helps to optimize the temporal attention scores.
  arXiv  Detail & Related papers  (2022-09-08T07:20:51Z)
• NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions [80.60423934589515]
  We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark. We set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. We find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning.
  arXiv  Detail & Related papers  (2021-05-18T04:56:46Z)
• End-to-End Video Question-Answer Generation with Generator-Pretester Network [27.31969951281815]
  We study a novel task, Video Question-Answer Generation (VQAG) for challenging Video Question Answering (Video QA) task in multimedia. As captions neither fully represent a video, nor are they always practically available, it is crucial to generate question-answer pairs based on a video via Video Question-Answer Generation (VQAG) We evaluate our system with the only two available large-scale human-annotated Video QA datasets and achieves state-of-the-art question generation performances.
  arXiv  Detail & Related papers  (2021-01-05T10:46:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.