Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media

• URL: http://arxiv.org/abs/2306.05115v1
• Date: Thu, 8 Jun 2023 11:29:58 GMT
• Title: Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media
• Authors: Thales Bertaglia, Stefan Huber, Catalina Goanta, Gerasimos Spanakis, Adriana Iamnitchi
• Abstract summary: Regulatory bodies worldwide are intensifying their efforts to ensure transparency in influencer marketing on social media. The task of automatically detecting sponsored content aims to enable the monitoring and enforcement of such regulations at scale. We propose using chatGPT to augment the annotation process with phrases identified as relevant features and brief explanations.
• Score: 4.322339935902437
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Regulatory bodies worldwide are intensifying their efforts to ensure transparency in influencer marketing on social media through instruments like the Unfair Commercial Practices Directive (UCPD) in the European Union, or Section 5 of the Federal Trade Commission Act. Yet enforcing these obligations has proven to be highly problematic due to the sheer scale of the influencer market. The task of automatically detecting sponsored content aims to enable the monitoring and enforcement of such regulations at scale. Current research in this field primarily frames this problem as a machine learning task, focusing on developing models that achieve high classification performance in detecting ads. These machine learning tasks rely on human data annotation to provide ground truth information. However, agreement between annotators is often low, leading to inconsistent labels that hinder the reliability of models. To improve annotation accuracy and, thus, the detection of sponsored content, we propose using chatGPT to augment the annotation process with phrases identified as relevant features and brief explanations. Our experiments show that this approach consistently improves inter-annotator agreement and annotation accuracy. Additionally, our survey of user experience in the annotation task indicates that the explanations improve the annotators' confidence and streamline the process. Our proposed methods can ultimately lead to more transparency and alignment with regulatory requirements in sponsored content detection.

Related papers

• From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants [0.0]
  Building reliable conversational AI assistants for customer-facing industries remains challenging due to noisy conversational data, fragmented knowledge, and the requirement for accurate human hand-off.<n>This paper presents an end-to-end framework for constructing and evaluating a conversational AI assistant directly from historical call transcripts.
  arXiv  Detail & Related papers  (2026-01-26T07:44:47Z)
• From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models [77.04403907729738]
  This survey charts the evolution of uncertainty from a passive diagnostic metric to an active control signal guiding real-time model behavior.<n>We demonstrate how uncertainty is leveraged as an active control signal across three frontiers.<n>This survey argues that mastering the new trend of uncertainty is essential for building the next generation of scalable, reliable, and trustworthy AI.
  arXiv  Detail & Related papers  (2026-01-22T06:21:31Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• ORCHID: Orchestrated Retrieval-Augmented Classification with Human-in-the-Loop Intelligent Decision-Making for High-Risk Property [6.643427585499247]
  ORCHID is a modular agentic system for HRP classification.<n>It pairs retrieval-augmented generation (RAG) with human oversight to produce policy-based outputs that can be audited.<n>The demonstration shows single item submission, grounded citations, SME feedback capture, and exportable audit artifacts.
  arXiv  Detail & Related papers  (2025-11-07T03:48:05Z)
• Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption [94.887133335656]
  We revisit three classes of watermarking through this lens.<n>emphLLM text watermarking offers modest provider benefit when framed solely as an anti-misuse tool.<n>emphIn-context watermarking (ICW) is tailored for trusted parties, such as conference organizers or educators.
  arXiv  Detail & Related papers  (2025-10-21T06:34:51Z)
• Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrieval [1.515687944002438]
  The rapid spread of misinformation in the digital era poses significant challenges to public discourse.<n>Traditional human-led fact-checking methods, while credible, struggle with the volume and velocity of online content.<n>This paper proposes a novel multi-agent system for automated fact-checking that enhances accuracy, efficiency, and explainability.
  arXiv  Detail & Related papers  (2025-06-22T02:39:27Z)
• Information Bargaining: Bilateral Commitment in Bayesian Persuasion [60.3761154043329]
  We introduce a unified framework and a well-structured solution concept for long-term persuasion.<n>This perspective makes explicit the common knowledge of the game structure and grants the receiver comparable commitment capabilities.<n>The framework is validated through a two-stage validation-and-inference paradigm.
  arXiv  Detail & Related papers  (2025-06-06T08:42:34Z)
• Engineering Trustworthy Machine-Learning Operations with Zero-Knowledge Proofs [1.7723990552388873]
  Zero-Knowledge Proofs (ZKPs) offer a cryptographic solution that enables provers to demonstrate, through verified computations, adherence to set requirements without revealing sensitive model details or data.<n>We identify five key properties (non-interactivity, transparent setup, standard representations, succinctness, and post-quantum security) critical for their application in AI validation and verification pipelines.
  arXiv  Detail & Related papers  (2025-05-26T15:39:11Z)
• Explainable Procedural Mistake Detection [27.40806437649092]
  Procedural mistake detection is a challenging sub-problem of classifying whether a human user has successfully executed the task at hand. We recast PMD to an explanatory self-dialog of questions and answers, which serve as evidence for a decision. Our results show that while open-source VLMs struggle with this task off-the-shelf, their accuracy, coherence, and dialog efficiency can be vastly improved.
  arXiv  Detail & Related papers  (2024-12-16T16:13:55Z)
• Evaluating Fairness in Transaction Fraud Models: Fairness Metrics, Bias Audits, and Challenges [3.499319293058353]
  Despite extensive research on algorithmic fairness, there is a notable gap in the study of bias in fraud detection models. These challenges include the need for fairness metrics that account for fraud data's imbalanced nature and the tradeoff between fraud protection and service quality. We present a comprehensive fairness evaluation of transaction fraud models using public synthetic datasets.
  arXiv  Detail & Related papers  (2024-09-06T16:08:27Z)
• LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems [16.546017147593044]
  This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility.
  arXiv  Detail & Related papers  (2024-03-14T18:59:10Z)
• Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy [0.0]
  The global surge in AI applications is transforming industries, leading to displacement and complementation of existing jobs, while also giving rise to new employment opportunities. Data annotation, encompassing the labelling of images or annotating of texts by human workers, crucially influences the quality of a dataset directly influences the quality of AI models trained on it. This paper delves into the economics of data annotation, with a specific focus on the impact of task instruction design and monetary incentives on data quality and costs.
  arXiv  Detail & Related papers  (2023-12-22T09:50:57Z)
• Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation [11.323961700172175]
  FACT-GPT is a framework designed to automate the claim matching phase of fact-checking using Large Language Models. This framework identifies new social media content that either supports or contradicts claims previously debunked by fact-checkers. We evaluated FACT-GPT on an extensive dataset of social media content related to public health.
  arXiv  Detail & Related papers  (2023-10-13T16:21:07Z)
• Fair-CDA: Continuous and Directional Augmentation for Group Fairness [48.84385689186208]
  We propose a fine-grained data augmentation strategy for imposing fairness constraints. We show that group fairness can be achieved by regularizing the models on transition paths of sensitive features between groups. Our proposed method does not assume any data generative model and ensures good generalization for both accuracy and fairness.
  arXiv  Detail & Related papers  (2023-04-01T11:23:00Z)
• ADEPT: A DEbiasing PrompT Framework [49.582497203415855]
  Finetuning is an applicable approach for debiasing contextualized word embeddings. discrete prompts with semantic meanings have shown to be effective in debiasing tasks. We propose ADEPT, a method to debias PLMs using prompt tuning while maintaining the delicate balance between removing biases and ensuring representation ability.
  arXiv  Detail & Related papers  (2022-11-10T08:41:40Z)
• Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach [131.20444904674494]
  We tackle the problem of measuring group fairness under unawareness of sensitive attributes. We show that quantification approaches are particularly suited to tackle the fairness-under-unawareness problem.
  arXiv  Detail & Related papers  (2021-09-17T13:45:46Z)
• FLAVA: Find, Localize, Adjust and Verify to Annotate LiDAR-Based Point Clouds [93.3595555830426]
  We propose FLAVA, a systematic approach to minimizing human interaction in the annotation process. Specifically, we divide the annotation pipeline into four parts: find, localize, adjust and verify. Our system also greatly reduces the amount of interaction by introducing a light-weight yet effective mechanism to propagate the results.
  arXiv  Detail & Related papers  (2020-11-20T02:22:36Z)
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty [66.17147341354577]
  We argue for considering a complementary form of transparency by estimating and communicating the uncertainty associated with model predictions. We describe how uncertainty can be used to mitigate model unfairness, augment decision-making, and build trustworthy systems. This work constitutes an interdisciplinary review drawn from literature spanning machine learning, visualization/HCI, design, decision-making, and fairness.
  arXiv  Detail & Related papers  (2020-11-15T17:26:14Z)
• Improving Classification through Weak Supervision in Context-specific Conversational Agent Development for Teacher Education [1.215785021723604]
  The effort required to develop an educational scenario specific conversational agent is time consuming. Previous approaches to modeling annotations have relied on labeling thousands of examples and calculating inter-annotator agreement and majority votes. We propose using a multi-task weak supervision method combined with active learning to address these concerns.
  arXiv  Detail & Related papers  (2020-10-23T23:39:40Z)
• Foreseeing the Benefits of Incidental Supervision [83.08441990812636]
  This paper studies whether we can, in a single framework, quantify the benefits of various types of incidental signals for a given target task without going through experiments. We propose a unified PAC-Bayesian motivated informativeness measure, PABI, that characterizes the uncertainty reduction provided by incidental supervision signals.
  arXiv  Detail & Related papers  (2020-06-09T20:59:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.