The Document Vectors Using Cosine Similarity Revisited
- URL: http://arxiv.org/abs/2205.13357v1
- Date: Thu, 26 May 2022 13:36:54 GMT
- Title: The Document Vectors Using Cosine Similarity Revisited
- Abstract summary: The current state-of-the-art test accuracy (97.42%) on the IMDB movie reviews dataset was reported by citetthongtan-phienthrakul 2019-sentiment.
We show that the previously reported test accuracy of 97.42% is invalid and should be corrected to 93.68%.
- Score: 0.48733623015338234
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The current state-of-the-art test accuracy (97.42\%) on the IMDB movie
reviews dataset was reported by \citet{thongtan-phienthrakul-2019-sentiment}
and achieved by the logistic regression classifier trained on the Document
Vectors using Cosine Similarity (DV-ngrams-cosine) proposed in their paper and
the Bag-of-N-grams (BON) vectors scaled by Naive Bayesian weights. While large
pre-trained Transformer-based models have shown SOTA results across many
datasets and tasks, the aforementioned model has not been surpassed by them,
despite being much simpler and pre-trained on the IMDB dataset only.
In this paper, we describe an error in the evaluation procedure of this
model, which was found when we were trying to analyze its excellent performance
on the IMDB dataset. We further show that the previously reported test accuracy
of 97.42\% is invalid and should be corrected to 93.68\%. We also analyze the
model performance with different amounts of training data (subsets of the IMDB
dataset) and compare it to the Transformer-based RoBERTa model. The results
show that while RoBERTa has a clear advantage for larger training sets, the
DV-ngrams-cosine performs better than RoBERTa when the labelled training set is
very small (10 or 20 documents). Finally, we introduce a sub-sampling scheme
based on Naive Bayesian weights for the training process of the
DV-ngrams-cosine, which leads to faster training and better quality.
Related papers
- Extracting alignment data in open models [50.81383232591576]
We show that it is possible to extract significant amounts of alignment training data from a post-trained model.<n>This data is useful to steer the model to improve certain capabilities such as long-context reasoning, safety, instruction following, and maths.<n>We find that models readily regurgitate training data that was used in post-training phases such as SFT or RL.
arXiv Detail & Related papers (2025-10-21T12:06:00Z) - Efficient Grammatical Error Correction Via Multi-Task Training and
Optimized Training Schedule [55.08778142798106]
We propose auxiliary tasks that exploit the alignment between the original and corrected sentences.
We formulate each task as a sequence-to-sequence problem and perform multi-task training.
We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
arXiv Detail & Related papers (2023-11-20T14:50:12Z) - Alleviating the Effect of Data Imbalance on Adversarial Training [26.36714114672729]
We study adversarial training on datasets that obey the long-tailed distribution.
We propose a new adversarial training framework -- Re-balancing Adversarial Training (REAT)
arXiv Detail & Related papers (2023-07-14T07:01:48Z) - Effective Robustness against Natural Distribution Shifts for Models with
Different Training Data [113.21868839569]
"Effective robustness" measures the extra out-of-distribution robustness beyond what can be predicted from the in-distribution (ID) performance.
We propose a new evaluation metric to evaluate and compare the effective robustness of models trained on different data.
arXiv Detail & Related papers (2023-02-02T19:28:41Z) - You Only Need a Good Embeddings Extractor to Fix Spurious Correlations [26.23962870932271]
GroupDRO requires training a model in an end-to-end manner with subgroup labels.
We show that we can achieve up to 90% accuracy without using any sub-group information in the training set.
arXiv Detail & Related papers (2022-12-12T21:42:33Z) - SynBench: Task-Agnostic Benchmarking of Pretrained Representations using
Synthetic Data [78.21197488065177]
Recent success in fine-tuning large models, that are pretrained on broad data at scale, on downstream tasks has led to a significant paradigm shift in deep learning.
This paper proposes a new task-agnostic framework, textitSynBench, to measure the quality of pretrained representations using synthetic data.
arXiv Detail & Related papers (2022-10-06T15:25:00Z) - Learning to be a Statistician: Learned Estimator for Number of Distinct
Values [54.629042119819744]
Estimating the number of distinct values (NDV) in a column is useful for many tasks in database systems.
In this work, we focus on how to derive accurate NDV estimations from random (online/offline) samples.
We propose to formulate the NDV estimation task in a supervised learning framework, and aim to learn a model as the estimator.
arXiv Detail & Related papers (2022-02-06T15:42:04Z) - GI-NNet \& RGI-NNet: Development of Robotic Grasp Pose Models, Trainable
with Large as well as Limited Labelled Training Datasets, under supervised
and semi supervised paradigms [0.0]
We use deep learning techniques to help robots learn to generate and execute appropriate grasps quickly.
We developed a Generative Inception Neural Network (GI-NNet) model, capable of generating antipodal robotic grasps on seen as well as unseen objects.
arXiv Detail & Related papers (2021-07-15T16:55:49Z) - Comparing Test Sets with Item Response Theory [53.755064720563]
We evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples.
We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models.
We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.
arXiv Detail & Related papers (2021-06-01T22:33:53Z) - Evaluating Prediction-Time Batch Normalization for Robustness under
Covariate Shift [81.74795324629712]
We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift.
We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness.
The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
arXiv Detail & Related papers (2020-06-19T05:08:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.