AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval

• URL: http://arxiv.org/abs/2203.16062v1
• Date: Wed, 30 Mar 2022 05:19:36 GMT
• Title: AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval
• Authors: Riku Togashi, Mayu Otani, Yuta Nakashima, Esa Rahtu, Janne Heikkila, Tetsuya Sakai
• Abstract summary: We propose an alternative measure for evaluating Video Moment Retrieval (VMR) We show that AxIoU satisfies two important axioms for VMR evaluation. We also empirically examine how AxIoU agrees with R@$K,theta$, as well as its stability with respect to change in the test data and human-annotated temporal boundaries.
• Score: 47.665259947270336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluation measures have a crucial impact on the direction of research. Therefore, it is of utmost importance to develop appropriate and reliable evaluation measures for new applications where conventional measures are not well suited. Video Moment Retrieval (VMR) is one such application, and the current practice is to use R@$K,\theta$ for evaluating VMR systems. However, this measure has two disadvantages. First, it is rank-insensitive: It ignores the rank positions of successfully localised moments in the top-$K$ ranked list by treating the list as a set. Second, it binarizes the Intersection over Union (IoU) of each retrieved video moment using the threshold $\theta$ and thereby ignoring fine-grained localisation quality of ranked moments. We propose an alternative measure for evaluating VMR, called Average Max IoU (AxIoU), which is free from the above two problems. We show that AxIoU satisfies two important axioms for VMR evaluation, namely, \textbf{Invariance against Redundant Moments} and \textbf{Monotonicity with respect to the Best Moment}, and also that R@$K,\theta$ satisfies the first axiom only. We also empirically examine how AxIoU agrees with R@$K,\theta$, as well as its stability with respect to change in the test data and human-annotated temporal boundaries.

Related papers

• Double Machine Learning for Conditional Moment Restrictions: IV Regression, Proximal Causal Learning and Beyond [16.842233444365764]
  Conditional moment restrictions (CMRs) are a key problem considered in statistics, causal inference, and econometrics.<n>Most CMR estimators use a two-stage approach, where the first-stage estimation is directly plugged into the second stage to estimate the function of interest.<n>We propose DML-CMR, a two-stage CMR estimator that provides an unbiased estimate with fast convergence rate guarantees.
  arXiv  Detail & Related papers  (2025-06-17T20:00:34Z)
• Rethinking Metrics and Benchmarks of Video Anomaly Detection [12.500876355560184]
  Video Anomaly Detection (VAD) aims to detect anomalies that deviate from expectation.<n>In this paper, we rethink VAD evaluation protocols through comprehensive experimental analyses.<n>We propose three novel evaluation methods to address these limitations.
  arXiv  Detail & Related papers  (2025-05-25T08:09:42Z)
• Near-Optimal Dynamic Regret for Adversarial Linear Mixture MDPs [63.47351876442425]
  We study episodic linear mixture MDPs with the unknown transition and adversarial rewards under full-information feedback. We propose a novel algorithm that combines the benefits of two popular methods: occupancy-measure-based and policy-based. Our algorithm enjoys an $widetildemathcalO(d sqrtH3 K + sqrtHK(H + barP_K$)$ dynamic regret, where $d$ is the feature dimension.
  arXiv  Detail & Related papers  (2024-11-05T13:55:52Z)
• Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes [44.974100402600165]
  We study the evaluation of a policy best-parametric and worst-case perturbations to a decision process (MDP) We use transition observations from the original MDP, whether they are generated under the same or a different policy. Our estimator is also estimated statistical inference using Wald confidence intervals.
  arXiv  Detail & Related papers  (2024-03-29T18:11:49Z)
• Faster Video Moment Retrieval with Point-Level Supervision [70.51822333023145]
  Video Moment Retrieval (VMR) aims at retrieving the most relevant events from an untrimmed video with natural language queries. Existing VMR methods suffer from two defects: massive expensive temporal annotations and complicated cross-modal interaction modules. We propose a novel method termed Cheaper and Faster Moment Retrieval (CFMR)
  arXiv  Detail & Related papers  (2023-05-23T12:53:50Z)
• Minimum mean-squared error estimation with bandit feedback [10.660855209170586]
  We consider the problem of sequentially learning to estimate, in the mean squared error (MSE) sense.<n>We propose two MSE estimators, and analyze their concentration properties.
  arXiv  Detail & Related papers  (2022-03-31T05:33:32Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• TISE: A Toolbox for Text-to-Image Synthesis Evaluation [9.092600296992925]
  We conduct a study on state-of-the-art methods for single- and multi-object text-to-image synthesis. We propose a common framework for evaluating these methods.
  arXiv  Detail & Related papers  (2021-12-02T16:39:35Z)
• Universal Off-Policy Evaluation [64.02853483874334]
  We take the first steps towards a universal off-policy estimator (UnO) We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns.
  arXiv  Detail & Related papers  (2021-04-26T18:54:31Z)
• Maximum Moment Restriction for Instrumental Variable Regression [23.785259180682004]
  We propose a simple framework for nonlinear instrumental variable (IV) regression based on a kernelized conditional moment restriction (CMR) The MMR is formulated by maximizing the interaction between the residual and the instruments belonging to a unit ball in a kernel reproducing space (RKHS) We demonstrate the advantages of our framework over existing ones using experiments on both synthetic and real-world data.
  arXiv  Detail & Related papers  (2020-10-15T12:02:25Z)
• Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification [54.22421582955454]
  We provide the first result of the optimal minimax guarantees for the excess risk for adversarially robust classification. Results are stated in terms of the Adversarial Signal-to-Noise Ratio (AdvSNR), which generalizes a similar notion for standard linear classification to the adversarial setting.
  arXiv  Detail & Related papers  (2020-06-29T21:06:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.