QActor: On-line Active Learning for Noisy Labeled Stream Data

• URL: http://arxiv.org/abs/2001.10399v1
• Date: Tue, 28 Jan 2020 15:13:21 GMT
• Title: QActor: On-line Active Learning for Noisy Labeled Stream Data
• Authors: Taraneh Younesian, Zilong Zhao, Amirmasoud Ghiassi, Robert Birke, Lydia Y. Chen
• Abstract summary: We propose QActor which combines the selection of supposedly clean samples via quality models and actively querying an oracle for the most informative true labels. QActor swiftly combines the merits of quality models for data filtering and oracle queries for cleaning the most informative data. A central feature of QActor is to dynamically adjust the query limit according to the learning loss for each data batch.
• Score: 10.814099534254922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Noisy labeled data is more a norm than a rarity for self-generated content that is continuously published on the web and social media. Due to privacy concerns and governmental regulations, such a data stream can only be stored and used for learning purposes in a limited duration. To overcome the noise in this on-line scenario we propose QActor which novel combines: the selection of supposedly clean samples via quality models and actively querying an oracle for the most informative true labels. While the former can suffer from low data volumes of on-line scenarios, the latter is constrained by the availability and costs of human experts. QActor swiftly combines the merits of quality models for data filtering and oracle queries for cleaning the most informative data. The objective of QActor is to leverage the stringent oracle budget to robustly maximize the learning accuracy. QActor explores various strategies combining different query allocations and uncertainty measures. A central feature of QActor is to dynamically adjust the query limit according to the learning loss for each data batch. We extensively evaluate different image datasets fed into the classifier that can be standard machine learning (ML) models or deep neural networks (DNN) with noise label ratios ranging between 30% and 80%. Our results show that QActor can nearly match the optimal accuracy achieved using only clean data at the cost of at most an additional 6% of ground truth data from the oracle.

Related papers

• Training on the Benchmark Is Not All You Need [52.01920740114261]
  We propose a simple and effective data leakage detection method based on the contents of multiple-choice options. Our method is able to work under black-box conditions without access to model training data or weights. We evaluate the degree of data leakage of 31 mainstream open-source LLMs on four benchmark datasets.
  arXiv  Detail & Related papers  (2024-09-03T11:09:44Z)
• Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
  We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books. Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
  arXiv  Detail & Related papers  (2024-05-31T20:15:10Z)
• Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic [99.3682210827572]
  Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. Data curation strategies are typically developed agnostic of the available compute for training. We introduce neural scaling laws that account for the non-homogeneous nature of web data.
  arXiv  Detail & Related papers  (2024-04-10T17:27:54Z)
• MeaeQ: Mount Model Extraction Attacks with Efficient Queries [6.1106195466129485]
  We study model extraction attacks in natural language processing (NLP) We propose MeaeQ, a straightforward yet effective method to address these issues. MeaeQ achieves higher functional similarity to the victim model than baselines while requiring fewer queries.
  arXiv  Detail & Related papers  (2023-10-21T16:07:16Z)
• Test-Time Self-Adaptive Small Language Models for Question Answering [63.91013329169796]
  We show and investigate the capabilities of smaller self-adaptive LMs, only with unlabeled test data. Our proposed self-adaption strategy demonstrates significant performance improvements on benchmark QA datasets.
  arXiv  Detail & Related papers  (2023-10-20T06:49:32Z)
• QASnowball: An Iterative Bootstrapping Framework for High-Quality Question-Answering Data Generation [67.27999343730224]
  We introduce an iterative bootstrapping framework for QA data augmentation (named QASnowball) QASnowball can iteratively generate large-scale high-quality QA data based on a seed set of supervised examples. We conduct experiments in the high-resource English scenario and the medium-resource Chinese scenario, and the experimental results show that the data generated by QASnowball can facilitate QA models.
  arXiv  Detail & Related papers  (2023-09-19T05:20:36Z)
• Knockoffs-SPR: Clean Sample Selection in Learning with Noisy Labels [56.81761908354718]
  We propose a novel theoretically guaranteed clean sample selection framework for learning with noisy labels. Knockoffs-SPR can be regarded as a sample selection module for a standard supervised training pipeline. We further combine it with a semi-supervised algorithm to exploit the support of noisy data as unlabeled data.
  arXiv  Detail & Related papers  (2023-01-02T07:13:28Z)
• Cost-Effective Online Contextual Model Selection [14.094350329970537]
  We formulate this task as an online contextual active model selection problem, where at each round the learner receives an unlabeled data point along with a context. The goal is to output the best model for any given context without obtaining an excessive amount of labels. We propose a contextual active model selection algorithm (CAMS), which relies on a novel uncertainty sampling query criterion defined on a given policy class for adaptive model selection.
  arXiv  Detail & Related papers  (2022-07-13T08:22:22Z)
• Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking [30.155625852894797]
  We propose a browser-based benchmarking tool for researchers and challenge organizers. Our tool helps test generalization capabilities of models across multiple datasets. Interactive filtering facilitates discovery of problematic behavior.
  arXiv  Detail & Related papers  (2021-10-11T11:08:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.