Stance in Replies and Quotes (SRQ): A New Dataset For Learning Stance in Twitter Conversations

• URL: http://arxiv.org/abs/2006.00691v2
• Date: Sat, 27 Jun 2020 21:23:20 GMT
• Title: Stance in Replies and Quotes (SRQ): A New Dataset For Learning Stance in Twitter Conversations
• Authors: Ramon Villa-Cox, Sumeet Kumar, Matthew Babcock, Kathleen M. Carley
• Abstract summary: We present the largest human-labeled stance dataset for Twitter conversations with over 5200 stance labels. We include many baseline models for learning the stance in conversations and compare the performance of various models.
• Score: 8.097870074875729
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated ways to extract stance (denying vs. supporting opinions) from conversations on social media are essential to advance opinion mining research. Recently, there is a renewed excitement in the field as we see new models attempting to improve the state-of-the-art. However, for training and evaluating the models, the datasets used are often small. Additionally, these small datasets have uneven class distributions, i.e., only a tiny fraction of the examples in the dataset have favoring or denying stances, and most other examples have no clear stance. Moreover, the existing datasets do not distinguish between the different types of conversations on social media (e.g., replying vs. quoting on Twitter). Because of this, models trained on one event do not generalize to other events. In the presented work, we create a new dataset by labeling stance in responses to posts on Twitter (both replies and quotes) on controversial issues. To the best of our knowledge, this is currently the largest human-labeled stance dataset for Twitter conversations with over 5200 stance labels. More importantly, we designed a tweet collection methodology that favors the selection of denial-type responses. This class is expected to be more useful in the identification of rumors and determining antagonistic relationships between users. Moreover, we include many baseline models for learning the stance in conversations and compare the performance of various models. We show that combining data from replies and quotes decreases the accuracy of models indicating that the two modalities behave differently when it comes to stance learning.

Related papers

• Multi-Class and Automated Tweet Categorization [0.0]
  The study reported here aims to detect the tweet category from its text. The tweet is categorized under 12 specified categories using Text Mining or Natural Language Processing (NLP), and Machine Learning (ML) techniques. The best ensemble model named, Gradient Boosting achieved an AUC score of 85%.
  arXiv  Detail & Related papers  (2021-11-13T14:28:47Z)
• Perceptual Score: What Data Modalities Does Your Model Perceive? [73.75255606437808]
  We introduce the perceptual score, a metric that assesses the degree to which a model relies on the different subsets of the input features. We find that recent, more accurate multi-modal models for visual question-answering tend to perceive the visual data less than their predecessors. Using the perceptual score also helps to analyze model biases by decomposing the score into data subset contributions.
  arXiv  Detail & Related papers  (2021-10-27T12:19:56Z)
• ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining [61.82562838486632]
  We crowdsource four new datasets on diverse online conversation forms of news comments, discussion forums, community question answering forums, and email threads. We benchmark state-of-the-art models on our datasets and analyze characteristics associated with the data.
  arXiv  Detail & Related papers  (2021-06-01T22:17:13Z)
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models [59.107260266206445]
  Short texts published on Twitter have earned significant attention as a rich source of information. Their inherent characteristics, such as the informal, and noisy linguistic style, remain challenging to many natural language processing (NLP) tasks. This study fulfils an assessment of existing language models in distinguishing the sentiment expressed in tweets by using a rich collection of 22 datasets.
  arXiv  Detail & Related papers  (2021-05-29T21:05:28Z)
• HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks [5.937482215664902]
  Social media content is often too noisy for direct use in any application. It is important to filter, categorize, and concisely summarize the available content to facilitate effective consumption and decision-making. We present a new large-scale dataset with 77K human-labeled tweets, sampled from a pool of 24 million tweets across 19 disaster events.
  arXiv  Detail & Related papers  (2021-04-07T12:29:36Z)
• A Weakly Supervised Approach for Classifying Stance in Twitter Replies [11.139350549173953]
  adversarial reactions are prevalent in online conversations. Inferring those adverse views (or stance) from the text in replies is difficult. We propose a weakly-supervised approach to predict the stance in Twitter replies.
  arXiv  Detail & Related papers  (2021-03-12T06:02:45Z)
• Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles [66.15398165275926]
  We propose a method that can automatically detect and ignore dataset-specific patterns, which we call dataset biases. Our method trains a lower capacity model in an ensemble with a higher capacity model. We show improvement in all settings, including a 10 point gain on the visual question answering dataset.
  arXiv  Detail & Related papers  (2020-11-07T22:20:03Z)
• Hidden Footprints: Learning Contextual Walkability from 3D Human Trails [70.01257397390361]
  Current datasets only tell you where people are, not where they could be. We first augment the set of valid, labeled walkable regions by propagating person observations between images, utilizing 3D information to create what we call hidden footprints. We devise a training strategy designed for such sparse labels, combining a class-balanced classification loss with a contextual adversarial loss.
  arXiv  Detail & Related papers  (2020-08-19T23:19:08Z)
• TIMME: Twitter Ideology-detection via Multi-task Multi-relational Embedding [26.074367752142198]
  We aim at solving the problem of predicting people's ideology, or political tendency. We estimate it by using Twitter data, and formalize it as a classification problem.
  arXiv  Detail & Related papers  (2020-06-02T00:00:39Z)
• The World is Not Binary: Learning to Rank with Grayscale Data for Dialogue Response Selection [55.390442067381755]
  We show that grayscale data can be automatically constructed without human effort. Our method employs off-the-shelf response retrieval models and response generation models as automatic grayscale data generators. Experiments on three benchmark datasets and four state-of-the-art matching models show that the proposed approach brings significant and consistent performance improvements.
  arXiv  Detail & Related papers  (2020-04-06T06:34:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.