InfFeed: Influence Functions as a Feedback to Improve the Performance of Subjective Tasks

• URL: http://arxiv.org/abs/2402.14702v2
• Date: Sat, 9 Mar 2024 19:13:54 GMT
• Title: InfFeed: Influence Functions as a Feedback to Improve the Performance of Subjective Tasks
• Authors: Somnath Banerjee, Maulindu Sarkar, Punyajoy Saha, Binny Mathew, Animesh Mukherjee
• Abstract summary: In this paper, we introduce InfFeed, which uses influence functions to compute the influential instances for a target instance. In doing this, InfFeed outperforms the state-of-the-art baselines by a maximum macro F1-score margin of almost 4% for hate speech classification. We also show that manually re-annotating only those silver annotated data points in the extension set that have a negative influence can immensely improve the model performance.
• Score: 10.124267937114611
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, influence functions present an apparatus for achieving explainability for deep neural models by quantifying the perturbation of individual train instances that might impact a test prediction. Our objectives in this paper are twofold. First we incorporate influence functions as a feedback into the model to improve its performance. Second, in a dataset extension exercise, using influence functions to automatically identify data points that have been initially `silver' annotated by some existing method and need to be cross-checked (and corrected) by annotators to improve the model performance. To meet these objectives, in this paper, we introduce InfFeed, which uses influence functions to compute the influential instances for a target instance. Toward the first objective, we adjust the label of the target instance based on its influencer(s) label. In doing this, InfFeed outperforms the state-of-the-art baselines (including LLMs) by a maximum macro F1-score margin of almost 4% for hate speech classification, 3.5% for stance classification, and 3% for irony and 2% for sarcasm detection. Toward the second objective we show that manually re-annotating only those silver annotated data points in the extension set that have a negative influence can immensely improve the model performance bringing it very close to the scenario where all the data points in the extension set have gold labels. This allows for huge reduction of the number of data points that need to be manually annotated since out of the silver annotated extension dataset, the influence function scheme picks up ~1/1000 points that need manual correction.

Related papers

• Stanceformer: Target-Aware Transformer for Stance Detection [59.69858080492586]
  Stance Detection involves discerning the stance expressed in a text towards a specific subject or target. Prior works have relied on existing transformer models that lack the capability to prioritize targets effectively. We introduce Stanceformer, a target-aware transformer model that incorporates enhanced attention towards the targets during both training and inference.
  arXiv  Detail & Related papers  (2024-10-09T17:24:28Z)
• Do Influence Functions Work on Large Language Models? [10.463762448166714]
  Influence functions aim to quantify the impact of individual training data points on a model's predictions. We evaluate influence functions across multiple tasks and find that they consistently perform poorly in most settings.
  arXiv  Detail & Related papers  (2024-09-30T06:50:18Z)
• Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs [1.5138606851862884]
  We present a framework that allows for maintaining generalizability and enhances the performance on the downstream task. We show that a linear transformation of the text representation from any transformer model using the task-specific concept operator results in a projection onto the latent concept space. Experimental results on three datasets, namely HateXplain, IMDB reviews, and Social Media Attributions, illustrate that the proposed model attains superior accuracy and generalizability.
  arXiv  Detail & Related papers  (2024-01-30T00:23:29Z)
• Influence Functions for Sequence Tagging Models [49.81774968547377]
  We extend influence functions to trace predictions back to the training points that informed them. We show the practical utility of segment influence by using the method to identify systematic annotation errors.
  arXiv  Detail & Related papers  (2022-10-25T17:13:11Z)
• Understanding Programmatic Weak Supervision via Source-aware Influence Function [76.74549130841383]
  Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels. We build on Influence Function (IF) to decompose the end model's training objective and then calculate the influence associated with each (data, source, class) These primitive influence score can then be used to estimate the influence of individual component PWS, such as source vote, supervision source, and training data.
  arXiv  Detail & Related papers  (2022-05-25T15:57:24Z)
• Function-words Enhanced Attention Networks for Few-Shot Inverse Relation Classification [17.078034236043738]
  We propose a function words adaptively enhanced attention framework (FAEA) for few-shot inverse relation classification. As the involvement of function words brings in significant intra-class redundancy, an adaptive message passing mechanism is introduced to capture and transfer inter-class differences. Our experimental results show that FAEA outperforms strong baselines, especially the inverse relation accuracy is improved by 14.33% under 1-shot setting in FewRel1.0.
  arXiv  Detail & Related papers  (2022-04-26T07:17:28Z)
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging [112.19994766375231]
  Influence functions approximate the 'influences' of training data-points for test predictions. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. Our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors.
  arXiv  Detail & Related papers  (2020-12-31T18:02:34Z)
• Towards Better Object Detection in Scale Variation with Adaptive Feature Selection [3.5352273012717044]
  We propose a novel adaptive feature selection module (AFSM) to automatically learn the way to fuse multi-level representations in the channel dimension. It significantly improves the performance of the detectors that have a feature pyramid structure. A class-aware sampling mechanism (CASM) is proposed to tackle the class imbalance problem.
  arXiv  Detail & Related papers  (2020-12-06T13:41:20Z)
• Deep F-measure Maximization for End-to-End Speech Understanding [52.36496114728355]
  We propose a differentiable approximation to the F-measure and train the network with this objective using standard backpropagation. We perform experiments on two standard fairness datasets, Adult, Communities and Crime, and also on speech-to-intent detection on the ATIS dataset and speech-to-image concept classification on the Speech-COCO dataset. In all four of these tasks, F-measure results in improved micro-F1 scores, with absolute improvements of up to 8% absolute, as compared to models trained with the cross-entropy loss function.
  arXiv  Detail & Related papers  (2020-08-08T03:02:27Z)
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions [55.660255727031725]
  Influence functions explain the decisions of a model by identifying influential training examples. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. We develop a new measure based on influence functions that can reveal artifacts in training data.
  arXiv  Detail & Related papers  (2020-05-14T00:45:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.