Understanding writing style in social media with a supervised contrastively pre-trained transformer

• URL: http://arxiv.org/abs/2310.11081v1
• Date: Tue, 17 Oct 2023 09:01:17 GMT
• Title: Understanding writing style in social media with a supervised contrastively pre-trained transformer
• Authors: Javier Huertas-Tato, Alejandro Martin, David Camacho
• Abstract summary: Online Social Networks serve as fertile ground for harmful behavior, ranging from hate speech to the dissemination of disinformation. We introduce the Style Transformer for Authorship Representations (STAR), trained on a large corpus derived from public sources of 4.5 x 106 authored texts. Using a support base of 8 documents of 512 tokens, we can discern authors from sets of up to 1616 authors with at least 80% accuracy.
• Score: 57.48690310135374
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online Social Networks serve as fertile ground for harmful behavior, ranging from hate speech to the dissemination of disinformation. Malicious actors now have unprecedented freedom to misbehave, leading to severe societal unrest and dire consequences, as exemplified by events such as the Capitol assault during the US presidential election and the Antivaxx movement during the COVID-19 pandemic. Understanding online language has become more pressing than ever. While existing works predominantly focus on content analysis, we aim to shift the focus towards understanding harmful behaviors by relating content to their respective authors. Numerous novel approaches attempt to learn the stylistic features of authors in texts, but many of these approaches are constrained by small datasets or sub-optimal training losses. To overcome these limitations, we introduce the Style Transformer for Authorship Representations (STAR), trained on a large corpus derived from public sources of 4.5 x 10^6 authored texts involving 70k heterogeneous authors. Our model leverages Supervised Contrastive Loss to teach the model to minimize the distance between texts authored by the same individual. This author pretext pre-training task yields competitive performance at zero-shot with PAN challenges on attribution and clustering. Additionally, we attain promising results on PAN verification challenges using a single dense layer, with our model serving as an embedding encoder. Finally, we present results from our test partition on Reddit. Using a support base of 8 documents of 512 tokens, we can discern authors from sets of up to 1616 authors with at least 80\% accuracy. We share our pre-trained model at huggingface (https://huggingface.co/AIDA-UPM/star) and our code is available at (https://github.com/jahuerta92/star)

Related papers

• COVID-19 Twitter Sentiment Classification Using Hybrid Deep Learning Model Based on Grid Search Methodology [0.0]
  The sentiment prediction is achieved using embedding, deep learning model and grid search algorithm on Twitter COVID-19 dataset. According to the study, public sentiment towards COVID-19 immunization appears to be improving with time.
  arXiv  Detail & Related papers  (2024-06-11T07:48:06Z)
• Breaking the Silence Detecting and Mitigating Gendered Abuse in Hindi, Tamil, and Indian English Online Spaces [0.6543929004971272]
  Team CNLP-NITS-PP developed an ensemble approach combining CNN and BiLSTM networks. CNN captures localized features indicative of abusive language through its convolution filters applied on embedded input text. BiLSTM analyzes this sequence for dependencies among words and phrases. validation scores showed strong performance across f1-measures, especially for English 0.84.
  arXiv  Detail & Related papers  (2024-04-02T14:55:47Z)
• Few-Shot Adversarial Prompt Learning on Vision-Language Models [62.50622628004134]
  The vulnerability of deep neural networks to imperceptible adversarial perturbations has attracted widespread attention. Previous efforts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. We propose a few-shot adversarial prompt framework where adapting input sequences with limited data makes significant adversarial robustness improvement.
  arXiv  Detail & Related papers  (2024-03-21T18:28:43Z)
• JAMDEC: Unsupervised Authorship Obfuscation using Constrained Decoding over Small Language Models [53.83273575102087]
  We propose an unsupervised inference-time approach to authorship obfuscation. We introduce JAMDEC, a user-controlled, inference-time algorithm for authorship obfuscation. Our approach builds on small language models such as GPT2-XL in order to help avoid disclosing the original content to proprietary LLM's APIs.
  arXiv  Detail & Related papers  (2024-02-13T19:54:29Z)
• Verifying the Robustness of Automatic Credibility Assessment [79.08422736721764]
  Text classification methods have been widely investigated as a way to detect content of low credibility. In some cases insignificant changes in input text can mislead the models. We introduce BODEGA: a benchmark for testing both victim models and attack methods on misinformation detection tasks.
  arXiv  Detail & Related papers  (2023-03-14T16:11:47Z)
• PART: Pre-trained Authorship Representation Transformer [64.78260098263489]
  Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)
• Data-Driven Mitigation of Adversarial Text Perturbation [1.3649494534428743]
  We propose a deobfuscation pipeline to make NLP models robust to adversarial text perturbations. We show CW2V embeddings are generally more robust to text perturbations than embeddings based on character ngrams. Our pipeline results in engagement bait classification that goes from 0.70 to 0.67 AUC with adversarial text perturbation.
  arXiv  Detail & Related papers  (2022-02-19T00:49:12Z)
• Evaluation of Deep Learning Models for Hostility Detection in Hindi Text [2.572404739180802]
  We present approaches for hostile text detection in the Hindi language. The proposed approaches are evaluated on the Constraint@AAAI 2021 Hindi hostility detection dataset. We evaluate a host of deep learning approaches based on CNN, LSTM, and BERT for this multi-label classification problem.
  arXiv  Detail & Related papers  (2021-01-11T19:10:57Z)
• Writer Identification Using Microblogging Texts for Social Media Forensics [53.180678723280145]
  We evaluate popular stylometric features, widely used in literary analysis, and specific Twitter features like URLs, hashtags, replies or quotes. We test varying sized author sets and varying amounts of training/test texts per author.
  arXiv  Detail & Related papers  (2020-07-31T00:23:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.