Hierarchical Bi-Directional Self-Attention Networks for Paper Review
Rating Recommendation
- URL: http://arxiv.org/abs/2011.00802v1
- Date: Mon, 2 Nov 2020 08:07:50 GMT
- Title: Hierarchical Bi-Directional Self-Attention Networks for Paper Review
Rating Recommendation
- Authors: Zhongfen Deng, Hao Peng, Congying Xia, Jianxin Li, Lifang He, Philip
S. Yu
- Abstract summary: We propose a Hierarchical bi-directional self-attention Network framework (HabNet) for paper review rating prediction and recommendation.
Specifically, we leverage the hierarchical structure of the paper reviews with three levels of encoders: sentence encoder (level one), intra-review encoder (level two) and inter-review encoder (level three)
We are able to identify useful predictors to make the final acceptance decision, as well as to help discover the inconsistency between numerical review ratings and text sentiment conveyed by reviewers.
- Score: 81.55533657694016
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Review rating prediction of text reviews is a rapidly growing technology with
a wide range of applications in natural language processing. However, most
existing methods either use hand-crafted features or learn features using deep
learning with simple text corpus as input for review rating prediction,
ignoring the hierarchies among data. In this paper, we propose a Hierarchical
bi-directional self-attention Network framework (HabNet) for paper review
rating prediction and recommendation, which can serve as an effective
decision-making tool for the academic paper review process. Specifically, we
leverage the hierarchical structure of the paper reviews with three levels of
encoders: sentence encoder (level one), intra-review encoder (level two) and
inter-review encoder (level three). Each encoder first derives contextual
representation of each level, then generates a higher-level representation, and
after the learning process, we are able to identify useful predictors to make
the final acceptance decision, as well as to help discover the inconsistency
between numerical review ratings and text sentiment conveyed by reviewers.
Furthermore, we introduce two new metrics to evaluate models in data imbalance
situations. Extensive experiments on a publicly available dataset (PeerRead)
and our own collected dataset (OpenReview) demonstrate the superiority of the
proposed approach compared with state-of-the-art methods.
Related papers
- Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework [0.1979158763744267]
Open-ended text generation has become a prominent task in natural language processing.
Decoding methods often excel in some metrics while underperforming in others.
We present novel ranking strategies within this multicriteria framework.
arXiv Detail & Related papers (2024-10-24T11:32:01Z) - On the Evaluation Consistency of Attribution-based Explanations [42.1421504321572]
We introduce Meta-Rank, an open platform for benchmarking attribution methods in the image domain.
Our benchmark reveals three insights in attribution evaluation endeavors: 1) evaluating attribution methods under disparate settings can yield divergent performance rankings; 2) although inconsistent across numerous cases, the performance rankings exhibit remarkable consistency across distinct checkpoints along the same training trajectory; and 3) prior attempts at consistent evaluation fare no better than baselines when extended to more heterogeneous models and datasets.
arXiv Detail & Related papers (2024-07-28T11:49:06Z) - Large Language Models are Diverse Role-Players for Summarization
Evaluation [82.31575622685902]
A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal.
Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions.
We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
arXiv Detail & Related papers (2023-03-27T10:40:59Z) - Investigating Fairness Disparities in Peer Review: A Language Model
Enhanced Approach [77.61131357420201]
We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs)
We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date.
We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
arXiv Detail & Related papers (2022-11-07T16:19:42Z) - LeQua@CLEF2022: Learning to Quantify [76.22817970624875]
LeQua 2022 is a new lab for the evaluation of methods for learning to quantify'' in textual datasets.
The goal of this lab is to provide a setting for the comparative evaluation of methods for learning to quantify, both in the binary setting and in the single-label multiclass setting.
arXiv Detail & Related papers (2021-11-22T14:54:20Z) - Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers [0.05857406612420462]
Large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks.
We propose evaluating systems through a novel measure of prediction coherence.
arXiv Detail & Related papers (2021-09-10T15:04:23Z) - Weakly-Supervised Aspect-Based Sentiment Analysis via Joint
Aspect-Sentiment Topic Embedding [71.2260967797055]
We propose a weakly-supervised approach for aspect-based sentiment analysis.
We learn sentiment, aspect> joint topic embeddings in the word embedding space.
We then use neural models to generalize the word-level discriminative information.
arXiv Detail & Related papers (2020-10-13T21:33:24Z) - A Unified Dual-view Model for Review Summarization and Sentiment
Classification with Inconsistency Loss [51.448615489097236]
Acquiring accurate summarization and sentiment from user reviews is an essential component of modern e-commerce platforms.
We propose a novel dual-view model that jointly improves the performance of these two tasks.
Experiment results on four real-world datasets from different domains demonstrate the effectiveness of our model.
arXiv Detail & Related papers (2020-06-02T13:34:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.