Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for
Simultaneous Speech Translation
- URL: http://arxiv.org/abs/2206.05807v2
- Date: Wed, 15 Jun 2022 18:31:15 GMT
- Title: Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for
Simultaneous Speech Translation
- Authors: Sara Papi, Marco Gaido, Matteo Negri, Marco Turchi
- Abstract summary: Simultaneous speech translation (SimulST) systems aim at generating their output with the lowest possible latency.
Average Lagging (AL) provides underestimated scores for systems that generate longer predictions compared to the corresponding references.
We show that this problem has practical relevance, as recent SimulST systems have indeed a tendency to over-generate.
- Score: 17.305879157385675
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Simultaneous speech translation (SimulST) systems aim at generating their
output with the lowest possible latency, which is normally computed in terms of
Average Lagging (AL). In this paper we highlight that, despite its widespread
adoption, AL provides underestimated scores for systems that generate longer
predictions compared to the corresponding references. We also show that this
problem has practical relevance, as recent SimulST systems have indeed a
tendency to over-generate. As a solution, we propose LAAL (Length-Adaptive
Average Lagging), a modified version of the metric that takes into account the
over-generation phenomenon and allows for unbiased evaluation of both
under-/over-generating systems.
Related papers
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution [87.3259169631789]
Nearest Speculative Decoding (NEST) is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources.
NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks.
In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B.
arXiv Detail & Related papers (2024-05-29T17:55:03Z) - Analytical Verification of Performance of Deep Neural Network Based
Time-Synchronized Distribution System State Estimation [0.18726646412385334]
Recently, we demonstrated success of a time-synchronized state estimator using deep neural networks (DNNs)
In this letter, we provide analytical bounds on the performance of that state estimator as a function of perturbations in the input measurements.
arXiv Detail & Related papers (2023-11-12T22:01:34Z) - Time-series Generation by Contrastive Imitation [87.51882102248395]
We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy.
At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
arXiv Detail & Related papers (2023-11-02T16:45:25Z) - Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for
Inverse Problem [97.64313409741614]
We prove fast mixing and characterize the stationary distribution of the Langevin Algorithm for inverting random weighted DNN generators.
We propose to do posterior sampling in the latent space of a pre-trained generative model.
arXiv Detail & Related papers (2022-06-18T03:47:37Z) - Low-rank Optimal Transport: Approximation, Statistics and Debiasing [51.50788603386766]
Low-rank optimal transport (LOT) approach advocated in citescetbon 2021lowrank
LOT is seen as a legitimate contender to entropic regularization when compared on properties of interest.
We target each of these areas in this paper in order to cement the impact of low-rank approaches in computational OT.
arXiv Detail & Related papers (2022-05-24T20:51:37Z) - Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming
Disfluency Detection [3.884530687475798]
Streaming BERT-based sequence tagging model is capable of detecting disfluencies in real-time.
Model attains state-of-the-art latency and stability scores when compared with recent work on incremental disfluency detection.
arXiv Detail & Related papers (2022-05-02T02:13:24Z) - Factual Error Correction for Abstractive Summaries Using Entity
Retrieval [57.01193722520597]
We propose an efficient factual error correction system RFEC based on entities retrieval post-editing process.
RFEC retrieves the evidence sentences from the original document by comparing the sentences with the target summary.
Next, RFEC detects the entity-level errors in the summaries by considering the evidence sentences and substitutes the wrong entities with the accurate entities from the evidence sentences.
arXiv Detail & Related papers (2022-04-18T11:35:02Z) - SALAD: Self-Adaptive Lightweight Anomaly Detection for Real-time
Recurrent Time Series [1.0437764544103274]
This paper introduces SALAD, which is a Self-Adaptive Lightweight Anomaly Detection approach based on a special type of recurrent neural networks called Long Short-Term Memory (LSTM)
Experiments based on two real-world open-source time series datasets demonstrate that SALAD outperforms five other state-of-the-art anomaly detection approaches in terms of detection accuracy.
In addition, the results also show that SALAD is lightweight and can be deployed on a commodity machine.
arXiv Detail & Related papers (2021-04-19T10:36:23Z) - Stream-level Latency Evaluation for Simultaneous Machine Translation [5.50178437495268]
Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications.
This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation.
arXiv Detail & Related papers (2021-04-18T11:16:17Z) - Non-Stationary Delayed Bandits with Intermediate Observations [10.538264213183076]
Online recommender systems often face long delays in receiving feedback, especially when optimizing for some long-term metrics.
We introduce the problem of non-stationary, delayed bandits with intermediate observations.
We develop an efficient algorithm based on UCRL, and prove sublinear regret guarantees for its performance.
arXiv Detail & Related papers (2020-06-03T09:27:03Z) - Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence
Lip-Reading [96.48553941812366]
Lip-reading aims to infer the speech content from the lip movement sequence.
Traditional learning process of seq2seq models suffers from two problems.
We propose a novel pseudo-convolutional policy gradient (PCPG) based method to address these two problems.
arXiv Detail & Related papers (2020-03-09T09:12:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.