Proposing Topic Models and Evaluation Frameworks for Analyzing Associations with External Outcomes: An Application to Leadership Analysis Using Large-Scale Corporate Review Data

• URL: http://arxiv.org/abs/2604.18919v2
• Date: Fri, 24 Apr 2026 01:43:54 GMT
• Title: Proposing Topic Models and Evaluation Frameworks for Analyzing Associations with External Outcomes: An Application to Leadership Analysis Using Large-Scale Corporate Review Data
• Abstract summary: This study focuses on leadership analysis using corporate review data.<n>It proposes a method leveraging large language models to generate topics that satisfy topic specificity and polarity stance consistency.<n>Using employee reviews from OpenWork, a major corporate review platform in Japan, the proposed method achieves improved interpretability, specificity, and polarity consistency.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Analyzing topics extracted from text data in relation to external outcomes is important across fields such as computational social science and organizational research. However, existing topic modeling methods struggle to simultaneously achieve interpretability, topic specificity (alignment with concrete actions or characteristics), and polarity stance consistency (absence of mixed positive and negative evaluations within a topic). Focusing on leadership analysis using corporate review data, this study proposes a method leveraging large language models to generate topics that satisfy these properties, along with an evaluation framework tailored to external outcome analysis. The framework explicitly incorporates topic specificity and polarity stance consistency as evaluation criteria and examines automated evaluation methods based on existing metrics. Using employee reviews from OpenWork, a major corporate review platform in Japan, the proposed method achieves improved interpretability, specificity, and polarity consistency compared to existing approaches. In analyses of external outcomes such as employee morale, it also produces topics with higher explanatory power. These results suggest that the proposed method and evaluation framework provide a generalized approach for topic analysis in applications involving external outcomes.

Related papers

• An Instrument to Evaluate Governance Proposals: AI Policy Analysis at Scale [4.570330752352129]
  This paper introduces a policy analysis framework for AI governance proposals in an evolving and contested regulatory landscape.<n>We use a mixed-methods approach that integrates qualitative insights from subject matter experts with computational text analysis to inform the design of policy attribute rubrics.<n>Rather than assessing policy effectiveness or desirability, the framework focuses on relevance and alignment across attributes.
  arXiv  Detail & Related papers  (2026-07-30T12:06:53Z)
• LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges [4.655159257282136]
  This survey offers a systems-level analysis of large language models (LLMs)-based scientific peer review.<n>We focus on two core evaluative functions: critique generation and score prediction.<n>We identify emerging risks, including prompt injection, data poisoning, retrieval vulnerabilities, and reward hacking.
  arXiv  Detail & Related papers  (2026-06-23T18:12:39Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• Capturing Opinion Shifts in Deliberative Discourse through Frequency-based Quantum deep learning methods [0.0]
  Deliberation plays a crucial role in shaping outcomes by weighing diverse perspectives before reaching decisions.<n>With recent advancements in Natural Language Processing, it has become possible to computationally model deliberation.<n>We present a comparative analysis of multiple NLP techniques to evaluate how effectively models interpret deliberative discourse.
  arXiv  Detail & Related papers  (2025-09-26T17:23:55Z)
• Objective Metrics for Evaluating Large Language Models Using External Data Sources [4.574672973076743]
  This paper proposes a framework for leveraging subjective metrics derived from the class textual materials across different semesters.<n>The framework emphasizes automation and transparency in scoring, reducing reliance on human interpretation.<n>This method addresses the limitations of subjective evaluation methods, providing a scalable solution for performance assessment in educational, scientific, and other high-stakes domains.
  arXiv  Detail & Related papers  (2025-08-01T02:24:19Z)
• TopicImpact: Improving Customer Feedback Analysis with Opinion Units for Topic Modeling and Star-Rating Prediction [0.29021840487584505]
  We improve the extraction of insights from customer reviews by restructuring the topic modelling pipeline to operate on opinion units.<n>The result is a heightened performance of the subsequent topic modeling, leading to coherent and interpretable topics.<n>By correlating the topics and sentiments with business metrics, such as star ratings, we can gain insights on how specific customer concerns impact business outcomes.
  arXiv  Detail & Related papers  (2025-07-16T09:19:26Z)
• Identifying Aspects in Peer Reviews [59.02879434536289]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework [61.38174427966444]
  Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios.<n>Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models.<n>We propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses.
  arXiv  Detail & Related papers  (2025-02-26T06:31:45Z)
• STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond [68.47402386668846]
  We introduce Structured Reasoning In Critical Text Assessment (STRICTA) to model text assessment as an explicit, step-wise reasoning process.<n>STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory.<n>We apply STRICTA to a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers.
  arXiv  Detail & Related papers  (2024-09-09T06:55:37Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Leveraging Large Language Models for NLG Evaluation: Advances and Challenges [57.88520765782177]
  Large Language Models (LLMs) have opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.
  arXiv  Detail & Related papers  (2024-01-13T15:59:09Z)
• Exploring the Power of Topic Modeling Techniques in Analyzing Customer Reviews: A Comparative Analysis [0.0]
  Machine learning and natural language processing algorithms have been deployed to analyze the vast amount of textual data available online. In this study, we examine and compare five frequently used topic modeling methods specifically applied to customer reviews. Our findings reveal that BERTopic consistently yield more meaningful extracted topics and achieve favorable results.
  arXiv  Detail & Related papers  (2023-08-19T08:18:04Z)
• JST-RR Model: Joint Modeling of Ratings and Reviews in Sentiment-Topic Prediction [2.3834926671238916]
  We propose a probabilistic model to accommodate both textual reviews and overall ratings. The proposed method can enhance the prediction accuracy of review data and achieve an effective detection of interpretable topics and sentiments.
  arXiv  Detail & Related papers  (2021-02-18T15:47:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.