Comparative Insights from 12 Machine Learning Models in Extracting Economic Ideology from Political Text

• URL: http://arxiv.org/abs/2501.09719v1
• Date: Thu, 16 Jan 2025 18:06:22 GMT
• Title: Comparative Insights from 12 Machine Learning Models in Extracting Economic Ideology from Political Text
• Authors: Jihed Ncib,
• Abstract summary: This study conducts a systematic assessment of the capabilities of 12 machine learning models and model variations in detecting economic ideology. The analysis assesses the performance of several generative, fine-tuned, and zero-shot models at the granular and aggregate levels.
• Score: 0.0
• License:
• Abstract: This study conducts a systematic assessment of the capabilities of 12 machine learning models and model variations in detecting economic ideology. As an evaluation benchmark, I use manifesto data spanning six elections in the United Kingdom and pre-annotated by expert and crowd coders. The analysis assesses the performance of several generative, fine-tuned, and zero-shot models at the granular and aggregate levels. The results show that generative models such as GPT-4o and Gemini 1.5 Flash consistently outperform other models against all benchmarks. However, they pose issues of accessibility and resource availability. Fine-tuning yielded competitive performance and offers a reliable alternative through domain-specific optimization. But its dependency on training data severely limits scalability. Zero-shot models consistently face difficulties with identifying signals of economic ideology, often resulting in negative associations with human coding. Using general knowledge for the domain-specific task of ideology scaling proved to be unreliable. Other key findings include considerable within-party variation, fine-tuning benefiting from larger training data, and zero-shot's sensitivity to prompt content. The assessments include the strengths and limitations of each model and derive best-practices for automated analyses of political content.

Related papers

• A Structured Reasoning Framework for Unbalanced Data Classification Using Probabilistic Models [1.6951945839990796]
  The paper studies a Markov network model for unbalanced data, aiming to solve the problems of classification bias and insufficient minority class recognition ability. The experimental results show that the Markov network performs well in indicators such as weighted accuracy, F1 score, and AUC-ROC. Future research can focus on efficient model training, structural optimization, and deep learning integration in large-scale unbalanced data environments.
  arXiv  Detail & Related papers  (2025-02-05T17:20:47Z)
• Linear Discriminant Analysis in Credit Scoring: A Transparent Hybrid Model Approach [9.88281854509076]
  We implement Linear Discriminant Analysis (LDA) as a feature reduction technique, which reduces the burden of the models complexity. Our hybrid model, XG-DNN, outperformed other models with the highest accuracy of 99.45% and a 99% F1 score with LDA. To interpret model decisions, we have applied 2 different explainable AI techniques named LIME (local) and Morris Sensitivity Analysis (global)
  arXiv  Detail & Related papers  (2024-12-05T14:21:18Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check [53.152011258252315]
  We show that using phonetic and graphic information reasonably is effective for Chinese Spelling Check. Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models. The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.
  arXiv  Detail & Related papers  (2023-07-25T17:02:38Z)
• Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes [72.13373216644021]
  We study the societal impact of machine learning by considering the collection of models that are deployed in a given context. We find deployed machine learning is prone to systemic failure, meaning some users are exclusively misclassified by all models available. These examples demonstrate ecosystem-level analysis has unique strengths for characterizing the societal impact of machine learning.
  arXiv  Detail & Related papers  (2023-07-12T01:11:52Z)
• Robustness Gym: Unifying the NLP Evaluation Landscape [91.80175115162218]
  Deep neural networks are often brittle when deployed in real-world systems. Recent research has focused on testing the robustness of such models. We propose a solution in the form of Robustness Gym, a simple and evaluation toolkit.
  arXiv  Detail & Related papers  (2021-01-13T02:37:54Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.