Topic Modeling on User Stories using Word Mover's Distance
- URL: http://arxiv.org/abs/2007.05302v2
- Date: Mon, 13 Jul 2020 09:22:50 GMT
- Title: Topic Modeling on User Stories using Word Mover's Distance
- Authors: Kim Julian G\"ulle, Nicholas Ford, Patrick Ebel, Florian Brokhausen,
Andreas Vogelsang
- Abstract summary: This paper focuses on topic modeling as a means to identify topics within a large set of crowd-generated user stories.
We evaluate the approaches on a publicly available set of 2,966 user stories written and categorized by crowd workers.
- Score: 4.378337862197529
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Requirements elicitation has recently been complemented with crowd-based
techniques, which continuously involve large, heterogeneous groups of users who
express their feedback through a variety of media. Crowd-based elicitation has
great potential for engaging with (potential) users early on but also results
in large sets of raw and unstructured feedback. Consolidating and analyzing
this feedback is a key challenge for turning it into sensible user
requirements. In this paper, we focus on topic modeling as a means to identify
topics within a large set of crowd-generated user stories and compare three
approaches: (1) a traditional approach based on Latent Dirichlet Allocation,
(2) a combination of word embeddings and principal component analysis, and (3)
a combination of word embeddings and Word Mover's Distance. We evaluate the
approaches on a publicly available set of 2,966 user stories written and
categorized by crowd workers. We found that a combination of word embeddings
and Word Mover's Distance is most promising. Depending on the word embeddings
we use in our approaches, we manage to cluster the user stories in two ways:
one that is closer to the original categorization and another that allows new
insights into the dataset, e.g. to find potentially new categories.
Unfortunately, no measure exists to rate the quality of our results
objectively. Still, our findings provide a basis for future work towards
analyzing crowd-sourced user stories.
Related papers
- CAST: Corpus-Aware Self-similarity Enhanced Topic modelling [16.562349140796115]
We introduce CAST: Corpus-Aware Self-similarity Enhanced Topic modelling, a novel topic modelling method.
We find self-similarity to be an effective metric to prevent functional words from acting as candidate topic words.
Our approach significantly enhances the coherence and diversity of generated topics, as well as the topic model's ability to handle noisy data.
arXiv Detail & Related papers (2024-10-19T15:27:11Z) - Annotator in the Loop: A Case Study of In-Depth Rater Engagement to Create a Bridging Benchmark Dataset [1.825224193230824]
We describe a novel, collaborative, and iterative annotator-in-the-loop methodology for annotation.
Our findings indicate that collaborative engagement with annotators can enhance annotation methods.
arXiv Detail & Related papers (2024-08-01T19:11:08Z) - Personalized Topic Selection Model for Topic-Grounded Dialogue [24.74527189182273]
Current models tend to predict user-uninteresting and contextually irrelevant topics.
We propose a textbfPersonalized topic stextbfElection model for textbfTopic-grounded textbfDialogue, named textbfPETD.
Our proposed method can generate engaging and diverse responses, outperforming state-of-the-art baselines.
arXiv Detail & Related papers (2024-06-04T06:09:49Z) - Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books.
Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
arXiv Detail & Related papers (2024-05-31T20:15:10Z) - UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset.
Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
arXiv Detail & Related papers (2023-10-02T17:40:01Z) - Large Language Models are Diverse Role-Players for Summarization
Evaluation [82.31575622685902]
A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal.
Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions.
We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
arXiv Detail & Related papers (2023-03-27T10:40:59Z) - Perceptual Score: What Data Modalities Does Your Model Perceive? [73.75255606437808]
We introduce the perceptual score, a metric that assesses the degree to which a model relies on the different subsets of the input features.
We find that recent, more accurate multi-modal models for visual question-answering tend to perceive the visual data less than their predecessors.
Using the perceptual score also helps to analyze model biases by decomposing the score into data subset contributions.
arXiv Detail & Related papers (2021-10-27T12:19:56Z) - Author Clustering and Topic Estimation for Short Texts [69.54017251622211]
We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document.
We also simultaneously cluster users, removing the need for post-hoc cluster estimation.
Our method performs as well as -- or better -- than traditional approaches to problems arising in short text.
arXiv Detail & Related papers (2021-06-15T20:55:55Z) - User Factor Adaptation for User Embedding via Multitask Learning [45.56193775870044]
We treat the user interest as domains and empirically examine how the user language can vary across the user factor.
We propose a user embedding model to account for the language variability of user interests via a multitask learning framework.
The model learns user language and its variations without human supervision.
arXiv Detail & Related papers (2021-02-22T15:21:01Z) - Weakly-Supervised Aspect-Based Sentiment Analysis via Joint
Aspect-Sentiment Topic Embedding [71.2260967797055]
We propose a weakly-supervised approach for aspect-based sentiment analysis.
We learn sentiment, aspect> joint topic embeddings in the word embedding space.
We then use neural models to generalize the word-level discriminative information.
arXiv Detail & Related papers (2020-10-13T21:33:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.