ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling

• URL: http://arxiv.org/abs/2602.20166v1
• Date: Mon, 09 Feb 2026 02:25:23 GMT
• Title: ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling
• Authors: Yongda Yu, Lei Zhang, Xinxin Guo, Minghui Yu, Zhengqi Zhuang, Guoping Rong, Haifeng Shen, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu,
• Abstract summary: ConceptRM builds a high-quality corpus to train a reflection model capable of effectively intercepting false alerts.<n>With only a small amount of expert annotations as anchors, ConceptRM creates datasets with varying noise ratios.<n>By analyzing the consensus decisions of these models, it effectively identifies reliable negative samples from a noisy dataset.
• Score: 10.245468083483006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In many applications involving intelligent agents, the overwhelming volume of alerts (mostly false) generated by the agents may desensitize users and cause them to overlook critical issues, leading to the so-called ''alert fatigue''. A common strategy is to train a reflection model as a filter to intercept false alerts with labelled data collected from user verification feedback. However, a key challenge is the noisy nature of such data as it is often collected in production environments. As cleaning noise via manual annotation incurs high costs, this paper proposes a novel method ConceptRM for constructing a high-quality corpus to train a reflection model capable of effectively intercepting false alerts. With only a small amount of expert annotations as anchors, ConceptRM creates perturbed datasets with varying noise ratios and utilizes co-teaching to train multiple distinct models for collaborative learning. By analyzing the consensus decisions of these models, it effectively identifies reliable negative samples from a noisy dataset. Experimental results demonstrate that ConceptRM significantly enhances the interception of false alerts with minimal annotation cost, outperforming several state-of-the-art LLM baselines by up to 53.31% on in-domain datasets and 41.67% on out-of-domain datasets.

Related papers

• Semantics-Aware Denoising: A PLM-Guided Sample Reweighting Strategy for Robust Recommendation [4.631922211808715]
  Implicit feedback, such as user clicks, serves as the primary data source for modern recommender systems.<n>We propose SAID (Semantics-Aware Implicit Denoising), a framework that leverages semantic consistency between user interests and item content to identify and downweight potentially noisy interactions.<n>Experiments on two real-world datasets demonstrate that SAID consistently improves recommendation performance, achieving up to 2.2% relative improvement in AUC over strong baselines.
  arXiv  Detail & Related papers  (2026-02-17T04:58:21Z)
• Noise as a Probe: Membership Inference Attacks on Diffusion Models Leveraging Initial Noise [51.179816451161635]
  Diffusion models have achieved remarkable progress in image generation, but their increasing deployment raises serious concerns about privacy.<n>In this work, we utilize a critical yet overlooked vulnerability: the widely used noise schedules fail to fully eliminate semantic information in the images.<n>We propose a simple yet effective membership inference attack, which injects semantic information into the initial noise and infers membership by analyzing the model's generation result.
  arXiv  Detail & Related papers  (2026-01-29T12:29:01Z)
• Lost in the Noise: How Reasoning Models Fail with Contextual Distractors [57.31788955167306]
  Recent advances in reasoning models and agentic AI systems have led to an increased reliance on diverse external information.<n>We introduce NoisyBench, a comprehensive benchmark that systematically evaluates model robustness across 11 datasets in RAG, reasoning, alignment, and tool-use tasks.<n>Our evaluation reveals a catastrophic performance drop of up to 80% in state-of-the-art models when faced with contextual distractors.
  arXiv  Detail & Related papers  (2026-01-12T05:43:51Z)
• SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models [53.19726629537694]
  Post-training alignment of video generation models with human preferences is a critical goal.<n>Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise.<n>We propose SoliReward, a systematic framework for video RM training.
  arXiv  Detail & Related papers  (2025-12-17T14:28:23Z)
• Learning to Retrieve with Weakened Labels: Robust Training under Label Noise [0.0]
  We consider a label weakening approach to generate robust retrieval models in the presence of label noise.<n>Our initial results show that label weakening can improve the performance of the retrieval tasks in comparison to 10 different state-of-the-art loss functions.
  arXiv  Detail & Related papers  (2025-12-15T11:52:13Z)
• Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances [55.37242480995541]
  We propose to denoise noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.
  arXiv  Detail & Related papers  (2023-10-25T17:23:37Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Learning with Noisy labels via Self-supervised Adversarial Noisy Masking [33.87292143223425]
  We propose a novel training approach termed adversarial noisy masking. It adaptively modulates the input data and label simultaneously, preventing the model to overfit noisy samples. It is tested on both synthetic and real-world noisy datasets.
  arXiv  Detail & Related papers  (2023-02-14T03:13:26Z)
• A Free Lunch to Person Re-identification: Learning from Automatically Generated Noisy Tracklets [52.30547023041587]
  unsupervised video-based re-identification (re-ID) methods have been proposed to solve the problem of high labor cost required to annotate re-ID datasets. But their performance is still far lower than the supervised counterparts. In this paper, we propose to tackle this problem by learning re-ID models from automatically generated person tracklets.
  arXiv  Detail & Related papers  (2022-04-02T16:18:13Z)
• Device-Directed Speech Detection: Regularization via Distillation for Weakly-Supervised Models [13.456066434598155]
  We address the problem of detecting speech directed to a device that does not contain a specific wake-word. Specifically, we focus on audio coming from a touch-based invocation.
  arXiv  Detail & Related papers  (2022-03-30T01:27:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.