Detecting Problem Statements in Peer Assessments

• URL: http://arxiv.org/abs/2006.04532v1
• Date: Sat, 30 May 2020 03:03:46 GMT
• Title: Detecting Problem Statements in Peer Assessments
• Abstract summary: We use over 18,000 review comments that were labeled by the reviewees as either detecting or not detecting a problem with the work. We deploy several traditional machine-learning models, as well as neural-network models using GloVe and BERT embeddings.
• Score: 12.993637157790143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Effective peer assessment requires students to be attentive to the deficiencies in the work they rate. Thus, their reviews should identify problems. But what ways are there to check that they do? We attempt to automate the process of deciding whether a review comment detects a problem. We use over 18,000 review comments that were labeled by the reviewees as either detecting or not detecting a problem with the work. We deploy several traditional machine-learning models, as well as neural-network models using GloVe and BERT embeddings. We find that the best performer is the Hierarchical Attention Network classifier, followed by the Bidirectional Gated Recurrent Units (GRU) Attention and Capsule model with scores of 93.1% and 90.5% respectively. The best non-neural network model was the support vector machine with a score of 89.71%. This is followed by the Stochastic Gradient Descent model and the Logistic Regression model with 89.70% and 88.98%.

Related papers

• Cybersecurity Detection Classification with Reasoning-enabled Language Models [50.72675435043546]
  A major issue in Security Operations Centers (SOCs) is alert fatigue.<n>Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat.<n>We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections.
  arXiv  Detail & Related papers  (2026-07-30T16:22:13Z)
• Benchmarking Agentic Review Systems [25.437512824197736]
  A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research.<n>We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline.<n>We show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important errors, and earn positive feedback from real users.
  arXiv  Detail & Related papers  (2026-06-18T03:30:06Z)
• KWBench: Measuring Unprompted Problem Recognition in Knowledge Work [0.0]
  KWBench is a benchmark for unprompted problem recognition in large language models.<n>It can identify a professional scenario before attempting to solve it.<n>It contains 223 tasks sourced from practitioners across acquisitions, contract negotiations, clinical pharmacy, organizational politics, fraud analysis, and incentive design.
  arXiv  Detail & Related papers  (2026-04-17T07:04:54Z)
• Large Language Models Often Know When They Are Being Evaluated [0.015534429177540245]
  We investigate whether frontier language models can accurately classify transcripts based on whether they originate from evaluations or real-world deployment.<n>We construct a benchmark of 1,000 prompts and transcripts from 61 distinct datasets.<n>Our results indicate that frontier models already exhibit a substantial, though not yet, level of evaluation-awareness.
  arXiv  Detail & Related papers  (2025-05-28T12:03:09Z)
• Leveraging Large Language Models for Suicide Detection on Social Media with Limited Labels [3.1399304968349186]
  This paper explores the use of Large Language Models (LLMs) to automatically detect suicidal content in text-based social media posts. We develop an ensemble approach involving prompting with Qwen2-72B-Instruct, and using fine-tuned models such as Llama3-8B, Llama3.1-8B, and Gemma2-9B. Experimental results show that the ensemble model significantly improves the detection accuracy, by 5% points compared with the individual models.
  arXiv  Detail & Related papers  (2024-10-06T14:45:01Z)
• Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits! [51.668411293817464]
  Industry practitioners care about small improvements in malware detection accuracy because their models are deployed to hundreds of millions of machines. Academic research is often restrained to public datasets on the order of ten thousand samples. We devise an approach to generate a benchmark of difficulty from a pool of available samples.
  arXiv  Detail & Related papers  (2023-12-25T21:25:55Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• A Large-scale Study on Unsupervised Outlier Model Selection: Evaluating the Internal Model Evaluation Strategies [12.698430560442123]
  We study the feasibility of employing internal model evaluation strategies for selecting a model for outlier detection. We evaluate 7 different strategies on their ability to discriminate between models w.r.t. detection performance, without using any labels.
  arXiv  Detail & Related papers  (2021-04-03T14:56:29Z)
• Semi-Supervised Noisy Student Pre-training on EfficientNet Architectures for Plant Pathology Classification [0.0]
  In this report, we investigate the problem of pathology classification using images of a single leaf. We explore the use of standard benchmark models such as VGG16, ResNet101, and DenseNet 161 to achieve a 0.945 score on the task. We introduce the state-of-the-art idea of semi-supervised Noisy Student training to the EfficientNet, resulting in significant improvements in both accuracy and convergence rate.
  arXiv  Detail & Related papers  (2020-12-01T08:34:03Z)
• RethinkCWS: Is Chinese Word Segmentation a Solved Task? [81.11161697133095]
  The performance of the Chinese Word (CWS) systems has gradually reached a plateau with the rapid development of deep neural networks. In this paper, we take stock of what we have achieved and rethink what's left in the CWS task.
  arXiv  Detail & Related papers  (2020-11-13T11:07:08Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Evaluation Toolkit For Robustness Testing Of Automatic Essay Scoring Systems [64.4896118325552]
  We evaluate the current state-of-the-art AES models using a model adversarial evaluation scheme and associated metrics. We find that AES models are highly overstable. Even heavy modifications(as much as 25%) with content unrelated to the topic of the questions do not decrease the score produced by the models.
  arXiv  Detail & Related papers  (2020-07-14T03:49:43Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.