Leveraging AI Predicted and Expert Revised Annotations in Interactive Segmentation: Continual Tuning or Full Training?

• URL: http://arxiv.org/abs/2402.19423v1
• Date: Thu, 29 Feb 2024 18:22:12 GMT
• Title: Leveraging AI Predicted and Expert Revised Annotations in Interactive Segmentation: Continual Tuning or Full Training?
• Authors: Tiezheng Zhang, Xiaoxi Chen, Chongyu Qu, Alan Yuille, Zongwei Zhou
• Abstract summary: Human experts revise the annotations predicted by AI, and in turn, AI improves its predictions by learning from these revised annotations. The risk of catastrophic forgetting--the AI tends to forget the previously learned classes if it is only retrained using the expert revised classes. This paper proposes Continual Tuning to address the problems from two perspectives: network design and data reuse.
• Score: 7.742968966681627
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive segmentation, an integration of AI algorithms and human expertise, premises to improve the accuracy and efficiency of curating large-scale, detailed-annotated datasets in healthcare. Human experts revise the annotations predicted by AI, and in turn, AI improves its predictions by learning from these revised annotations. This interactive process continues to enhance the quality of annotations until no major revision is needed from experts. The key challenge is how to leverage AI predicted and expert revised annotations to iteratively improve the AI. Two problems arise: (1) The risk of catastrophic forgetting--the AI tends to forget the previously learned classes if it is only retrained using the expert revised classes. (2) Computational inefficiency when retraining the AI using both AI predicted and expert revised annotations; moreover, given the dominant AI predicted annotations in the dataset, the contribution of newly revised annotations--often account for a very small fraction--to the AI training remains marginal. This paper proposes Continual Tuning to address the problems from two perspectives: network design and data reuse. Firstly, we design a shared network for all classes followed by class-specific networks dedicated to individual classes. To mitigate forgetting, we freeze the shared network for previously learned classes and only update the class-specific network for revised classes. Secondly, we reuse a small fraction of data with previous annotations to avoid over-computing. The selection of such data relies on the importance estimate of each data. The importance score is computed by combining the uncertainty and consistency of AI predictions. Our experiments demonstrate that Continual Tuning achieves a speed 16x greater than repeatedly training AI from scratch without compromising the performance.

Related papers

• Ensuring Computer Science Learning in the AI Era: Open Generative AI Policies and Assignment-Driven Written Quizzes [0.0]
  This paper presents an assessment model that permits the use of generative AI for take-home programming assignments.<n>To promote authentic learning, in-class, closed-book assessments are weighted more heavily than the assignments themselves.<n> Statistical analyses revealed no meaningful linear correlation between GenAI usage levels and assessment outcomes.
  arXiv  Detail & Related papers  (2026-01-16T17:02:44Z)
• Evaluating Generative AI for CS1 Code Grading: Direct vs Reverse Methods [0.0]
  This paper compares two AI-based grading techniques: textitDirect, where the AI model applies a rubric directly to student code, and textitReverse (a newly proposed approach), where the AI first fixes errors, then deduces a grade based on the nature and number of fixes.<n>We discuss the strengths and limitations of each approach, practical considerations for prompt design, and future directions for hybrid human-AI grading systems.
  arXiv  Detail & Related papers  (2025-11-17T01:38:06Z)
• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• General Scales Unlock AI Evaluation with Explanatory and Predictive Power [57.7995945974989]
  benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems. We introduce general scales for AI evaluation that can explain what common AI benchmarks really measure. Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate.
  arXiv  Detail & Related papers  (2025-03-09T01:13:56Z)
• What should an AI assessor optimise for? [57.96463917842822]
  An AI assessor is an external, ideally indepen-dent system that predicts an indicator, e.g., a loss value, of another AI system. Here we address the question: is it always optimal to train the assessor for the target metric? We experimentally explore this question for, respectively, regression losses and classification scores with monotonic and non-monotonic mappings.
  arXiv  Detail & Related papers  (2025-02-01T08:41:57Z)
• Embracing Massive Medical Data [8.458637345001758]
  We propose an online learning method that enables training AI from massive medical data. Our method identifies the most significant samples for the current AI model based on their data uniqueness and prediction uncertainty. Compared with prevalent training paradigms, our method not only improves data efficiency by enabling training on continual data streams, but also mitigates catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-07-05T17:50:30Z)
• Adaptation of XAI to Auto-tuning for Numerical Libraries [0.0]
  Explainable AI (XAI) technology is gaining prominence, aiming to streamline AI model development and alleviate the burden of explaining AI outputs to users. This research focuses on XAI for AI models when integrated into two different processes for practical numerical computations.
  arXiv  Detail & Related papers  (2024-05-12T09:00:56Z)
• Learning-Augmented Algorithms with Explicit Predictors [67.02156211760415]
  Recent advances in algorithmic design show how to utilize predictions obtained by machine learning models from past and present data. Prior research in this context was focused on a paradigm where the predictor is pre-trained on past data and then used as a black box. In this work, we unpack the predictor and integrate the learning problem it gives rise for within the algorithmic challenge.
  arXiv  Detail & Related papers  (2024-03-12T08:40:21Z)
• Learning to Defer in Congested Systems: The AI-Human Interplay [4.324474867341765]
  High-stakes applications rely on combining Artificial Intelligence (AI) and humans for responsive and reliable decision making.<n>In this paper, we introduce a model to capture such an AI-human interplay.<n>We propose a near-optimal learning algorithm that carefully balances the classification loss from a selectively sampled dataset.
  arXiv  Detail & Related papers  (2024-02-19T15:47:47Z)
• ASPEST: Bridging the Gap Between Active Learning and Selective Prediction [56.001808843574395]
  Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
  arXiv  Detail & Related papers  (2023-04-07T23:51:07Z)
• Active Learning for Abstractive Text Summarization [50.79416783266641]
  We propose the first effective query strategy for Active Learning in abstractive text summarization. We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
  arXiv  Detail & Related papers  (2023-01-09T10:33:14Z)
• Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine [5.126042819606137]
  We focus on three key methodological approaches that address some of the particular challenges in AI-driven medical decision making. Domain adaptation and transfer learning enable AI models to be trained and applied across multiple domains. Federated learning enables learning large-scale models without exposing sensitive personal health information.
  arXiv  Detail & Related papers  (2022-11-17T03:32:00Z)
• Evaluating Machine Unlearning via Epistemic Uncertainty [78.27542864367821]
  This work presents an evaluation of Machine Unlearning algorithms based on uncertainty. This is the first definition of a general evaluation of our best knowledge.
  arXiv  Detail & Related papers  (2022-08-23T09:37:31Z)
• Visual correspondence-based explanations improve AI robustness and human-AI team accuracy [7.969008943697552]
  We propose two novel architectures of self-interpretable image classifiers that first explain, and then predict. Our models consistently improve (by 1 to 4 points) on out-of-distribution (OOD) datasets. For the first time, we show that it is possible to achieve complementary human-AI team accuracy (i.e., that is higher than either AI-alone or human-alone) in ImageNet and CUB image classification tasks.
  arXiv  Detail & Related papers  (2022-07-26T10:59:42Z)
• How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis [93.37576644429578]
  This work establishes the first theoretical analysis for the known iterative self-training paradigm. We prove the benefits of unlabeled data in both training convergence and generalization ability. Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training.
  arXiv  Detail & Related papers  (2022-01-21T02:16:52Z)
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation [109.11580756757611]
  Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members' performances. We introduce a novel training criterion called DICE: it increases diversity by reducing spurious correlations among features.
  arXiv  Detail & Related papers  (2021-01-14T10:53:26Z)
• Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time [34.03150701567508]
  Adding auxiliary losses to the main objective function is a general way of encoding biases that can help networks learn better representations. In this work we take inspiration from textittransductive learning and note that after receiving an input, we can fine-tune our networks on any unsupervised loss. We formulate em meta-tailoring, a nested optimization similar to that in meta-learning, and train our models to perform well on the task objective after adapting them using an unsupervised loss.
  arXiv  Detail & Related papers  (2020-09-22T15:26:24Z)
• CLAN: Continuous Learning using Asynchronous Neuroevolution on Commodity Edge Devices [3.812706195714961]
  We build a prototype distributed system of Raspberry Pis communicating via WiFi running NeuroEvolutionary (NE) learning and inference. We evaluate the performance of such a collaborative system and detail the compute/communication characteristics of different arrangements of the system.
  arXiv  Detail & Related papers  (2020-08-27T01:49:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.