The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling

• URL: http://arxiv.org/abs/2307.15508v1
• Date: Fri, 28 Jul 2023 12:08:15 GMT
• Title: The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling
• Abstract summary: We formalise and edits and revisions in incremental sequence labelling. We apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks.
• Score: 14.846377138993645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Incremental dialogue model components produce a sequence of output prefixes based on incoming input. Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy. In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies. We then apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks, paving the road for better revision policies.

Related papers

• Interpretable reinforcement learning with decision-tree pruning [0.0]
  Reinforcement learning policies are difficult to inspect, but interpreting them is a prerequisite for trustworthiness.<n>We present a pruning process that simplifies such rule-based policies while preserving task performance.<n>We investigate this approach on classic control and MuJoCo benchmarks, where pruning traces reveal consistent interpretability improvements.
  arXiv  Detail & Related papers  (2026-08-07T12:14:33Z)
• Kintsugi: Learning Policies by Repairing Executable Knowledge Bases [31.289004484222847]
  Kintsugi represents task-level policy knowledge as composable typed entries.<n>Between rollouts, a tool-constrained editing loop diagnoses trajectory failures, localizes them to editable KB layers, and proposes candidate edits.<n>A deterministic verification gate admits an edit only when the candidate type-checks, the resulting KB executes, and focused validation success or trajectory-health metrics improve without violating protected-regression checks.
  arXiv  Detail & Related papers  (2026-05-10T11:51:18Z)
• Polaris: A Gödel Agent Framework for Small Language Models through Experience-Abstracted Policy Repair [0.9780193377233132]
  We introduce Polaris, a Gdel agent for compact models that performs policy repair via experience abstraction.<n>Unlike response level self correction or parameter tuning, Polaris makes policy level changes with small, auditable patches that persist in the policy.<n>As part of the loop, the agent engages in meta reasoning: it explains its errors, proposes concrete revisions to its own policy, and then updates the policy.
  arXiv  Detail & Related papers  (2026-03-24T12:25:32Z)
• Issue-Oriented Agent-Based Framework for Automated Review Comment Generation [15.04868140672973]
  RevAgent is a novel agent-based issue-oriented framework for code review comments.<n>It decomposes the task into three stages: Generation, Discrimination, and Training.<n>It significantly outperforms state-of-the-art PLM- and LLM-based baselines.
  arXiv  Detail & Related papers  (2025-11-01T11:44:11Z)
• A Planning Framework for Adaptive Labeling [8.883000217198843]
  We introduce an adaptive labeling framework where measurement effort can be reallocated in batches. We show that even one-step lookahead policy can substantially outperform common adaptive labelings. We propose a direct backpropagation-based approach, Smoothed-Autodiff, based on a carefully smoothed version of the original non-differentiable MDP.
  arXiv  Detail & Related papers  (2025-02-10T00:01:08Z)
• ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction [26.64363135181992]
  We explore the impact of shifting from sentence-level to paragraph-level scope for the task of scientific text revision. The paragraph level definition of the task allows for more meaningful changes, and is guided by detailed revision instructions rather than general ones. Our experiments demonstrate that using detailed instructions significantly improves the quality of automated revisions compared to general approaches.
  arXiv  Detail & Related papers  (2025-01-09T13:19:55Z)
• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing [19.812562421377706]
  In NLP, incremental processors produce output in instalments, based on incoming prefixes of the linguistic input. Some tokens trigger revisions, causing edits to the output hypothesis, but little is known about why models revise when they revise. We investigate the appropriateness of regressions and skips in human reading eye-tracking data as signals to inform revision policies in incremental sequence labelling.
  arXiv  Detail & Related papers  (2023-10-27T16:08:15Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translation [17.802607889752736]
  Simultaneous machine translation (SiMT) starts its translation before reading the whole source sentence. We propose a method of performing the adaptive policy via integrating post-evaluation into the fixed policy.
  arXiv  Detail & Related papers  (2022-10-21T11:57:14Z)
• Topic-Controllable Summarization: Topic-Aware Evaluation and Transformer Methods [4.211128681972148]
  Topic-controllable summarization is an emerging research area with a wide range of potential applications. This work proposes a new topic-oriented evaluation measure to automatically evaluate the generated summaries. In addition, we adapt topic embeddings to work with powerful Transformer architectures and propose a novel and efficient approach for guiding the summary generation through control tokens.
  arXiv  Detail & Related papers  (2022-06-09T07:28:16Z)
• TRUE: Re-evaluating Factual Consistency Evaluation [29.888885917330327]
  We introduce TRUE: a comprehensive study of factual consistency metrics on a standardized collection of existing texts from diverse tasks. Our standardization enables an example-level meta-evaluation protocol that is more actionable and interpretable than previously reported correlations. Across diverse state-of-the-art metrics and 11 datasets we find that large-scale NLI and question generation-and-answering-based approaches achieve strong and complementary results.
  arXiv  Detail & Related papers  (2022-04-11T10:14:35Z)
• SVIP: Sequence VerIfication for Procedures in Videos [68.07865790764237]
  We propose a novel sequence verification task that aims to distinguish positive video pairs performing the same action sequence from negative ones with step-level transformations. Such a challenging task resides in an open-set setting without prior action detection or segmentation. We collect a scripted video dataset enumerating all kinds of step-level transformations in chemical experiments.
  arXiv  Detail & Related papers  (2021-12-13T07:03:36Z)
• Editing a classifier by rewriting its prediction rules [133.5026383860842]
  We present a methodology for modifying the behavior of a classifier by directly rewriting its prediction rules. Our approach requires virtually no additional data collection and can be applied to a variety of settings, including adapting a model to new environments.
  arXiv  Detail & Related papers  (2021-12-02T06:40:37Z)
• Uncertainty-Aware Annotation Protocol to Evaluate Deformable Registration Algorithms [3.2845753359072125]
  We introduce a principled strategy for the construction of a gold standard in deformable registration. Our framework: (i) iteratively suggests the most informative location to annotate next, taking into account its redundancy with previous annotations; (ii) extends traditional pointwise annotations by accounting for the spatial uncertainty of each annotation; and (iii) naturally provides a new strategy for the evaluation of deformable registration algorithms.
  arXiv  Detail & Related papers  (2021-04-02T19:31:19Z)
• Learning to Update Natural Language Comments Based on Code Changes [48.829941738578086]
  We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications.
  arXiv  Detail & Related papers  (2020-04-25T15:37:46Z)
• Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading [96.48553941812366]
  Lip-reading aims to infer the speech content from the lip movement sequence. Traditional learning process of seq2seq models suffers from two problems. We propose a novel pseudo-convolutional policy gradient (PCPG) based method to address these two problems.
  arXiv  Detail & Related papers  (2020-03-09T09:12:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.