Hit ratio: An Evaluation Metric for Hashtag Recommendation

• URL: http://arxiv.org/abs/2010.01258v1
• Date: Sat, 3 Oct 2020 02:07:41 GMT
• Title: Hit ratio: An Evaluation Metric for Hashtag Recommendation
• Authors: Areej Alsini, Du Q. Huynh, Amitava Datta
• Abstract summary: We propose a new metric which we call hit ratio for hashtag recommendation. Most of the research in the area of hashtag recommendation have used classical metrics such as hit rate, precision, recall, and F1-score. A comparison of hit ratio with the classical evaluation metrics reveals their limitations.
• Score: 6.746400031322727
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hashtag recommendation is a crucial task, especially with an increase of interest in using social media platforms such as Twitter in the last decade. Hashtag recommendation systems automatically suggest hashtags to a user while writing a tweet. Most of the research in the area of hashtag recommendation have used classical metrics such as hit rate, precision, recall, and F1-score to measure the accuracy of hashtag recommendation systems. These metrics are based on the exact match of the recommended hashtags with their corresponding ground truth. However, it is not clear how adequate these metrics to evaluate hashtag recommendation. The research question that we are interested in seeking an answer is: are these metrics adequate for evaluating hashtag recommendation systems when the numbers of ground truth hashtags in tweets are highly variable? In this paper, we propose a new metric which we call hit ratio for hashtag recommendation. Extensive evaluation through hypothetical examples and real-world application across a range of hashtag recommendation models indicate that the hit ratio is a useful metric. A comparison of hit ratio with the classical evaluation metrics reveals their limitations.

Related papers

• RIGHT: Retrieval-augmented Generation for Mainstream Hashtag Recommendation [76.24205422163169]
  We propose RetrIeval-augmented Generative Mainstream HashTag Recommender (RIGHT) RIGHT consists of three components: 1) a retriever seeks relevant hashtags from the entire tweet-hashtags set; 2) a selector enhances mainstream identification by introducing global signals; and 3) a generator incorporates input tweets and selected hashtags to directly generate the desired hashtags. Our method achieves significant improvements over state-of-the-art baselines. Moreover, RIGHT can be easily integrated into large language models, improving the performance of ChatGPT by more than 10%.
  arXiv  Detail & Related papers  (2023-12-16T14:47:03Z)
• Analyzing Trendy Twitter Hashtags in the 2022 French Election [0.0]
  We propose a method for using semantic networks as user-level features for machine learning tasks. We conducted an experiment using a semantic network of 1037 Twitter hashtags from a corpus of 3.7 million tweets related to the 2022 French presidential election. Our semantic feature performs well with the regression with most emotions having $R2$ above 0.5.
  arXiv  Detail & Related papers  (2023-10-11T15:17:55Z)
• #REVAL: a semantic evaluation framework for hashtag recommendation [6.746400031322727]
  We propose a novel semantic evaluation framework for hashtag recommendation, called #REval. #REval includes an internal module referred to as BERTag, which automatically learns the hashtag embeddings. Our experiments on three large datasets show that #REval gave more meaningful hashtag synonyms for hashtag recommendation evaluation.
  arXiv  Detail & Related papers  (2023-05-24T07:10:56Z)
• Hashtag-Guided Low-Resource Tweet Classification [31.810562621519804]
  We propose a novel Hashtag-guided Tweet Classification model (HashTation) HashTation automatically generates meaningful hashtags for the input tweet to provide useful auxiliary signals for tweet classification. Experiments show that HashTation achieves significant improvements on seven low-resource tweet classification tasks.
  arXiv  Detail & Related papers  (2023-02-20T18:21:02Z)
• Tag-Aware Document Representation for Research Paper Recommendation [68.8204255655161]
  We propose a hybrid approach that leverages deep semantic representation of research papers based on social tags assigned by users. The proposed model is effective in recommending research papers even when the rating data is very sparse.
  arXiv  Detail & Related papers  (2022-09-08T09:13:07Z)
• Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand [117.62186420147563]
  We propose a generalization of leaderboards, bidimensional leaderboards (Billboards) Unlike conventional unidimensional leaderboards that sort submitted systems by predetermined metrics, a Billboard accepts both generators and evaluation metrics as competing entries. We demonstrate that a linear ensemble of a few diverse metrics sometimes substantially outperforms existing metrics in isolation.
  arXiv  Detail & Related papers  (2021-12-08T06:34:58Z)
• Attend and Select: A Segment Attention based Selection Mechanism for Microblog Hashtag Generation [69.73215951112452]
  A hashtag is formed by tokens or phrases that may originate from various fragmentary segments of the original text. We propose an end-to-end Transformer-based generation model which consists of three phases: encoding, segments-selection, and decoding. We introduce two large-scale hashtag generation datasets, which are newly collected from Chinese Weibo and English Twitter.
  arXiv  Detail & Related papers  (2021-06-06T15:13:58Z)
• A Survey on Deep Hashing Methods [52.326472103233854]
  Nearest neighbor search aims to obtain the samples in the database with the smallest distances from them to the queries. With the development of deep learning, deep hashing methods show more advantages than traditional methods. Deep supervised hashing is categorized into pairwise methods, ranking-based methods, pointwise methods and quantization. Deep unsupervised hashing is categorized into similarity reconstruction-based methods, pseudo-label-based methods and prediction-free self-supervised learning-based methods.
  arXiv  Detail & Related papers  (2020-03-04T08:25:15Z)
• On Identifying Hashtags in Disaster Twitter Data [55.17975121160699]
  We construct a unique dataset of disaster-related tweets annotated with hashtags useful for filtering actionable information. Using this dataset, we investigate Long Short Term Memory-based models within a Multi-Task Learning framework. The best performing model achieves an F1-score as high as 92.22%.
  arXiv  Detail & Related papers  (2020-01-05T22:37:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.