The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel

• URL: http://arxiv.org/abs/2606.23042v1
• Date: Mon, 22 Jun 2026 08:48:50 GMT
• Title: The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel
• Abstract summary: I describe the panel, an applicability rule that keeps a score of zero distinct from a blank, and a lens system that separates what an actor says from what it does.<n>I try to be plain about what the method can't do, including the human validation it still lacks, and I release the instrument and data in full.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most tools for measuring political positions, manifesto coding, expert surveys, text-scaling models, were built and validated on Western party systems, and outside that setting they work poorly, and often not at all. This paper is an attempt at a method for those settings. It treats a large language model not as a measurement device but as a single, fallible rater in a panel, roughly the way an expert survey treats one expert: the value comes from pooling many judges rather than trusting any one of them. I describe the panel, an applicability rule that keeps a score of zero distinct from a blank, and a lens system that separates what an actor says from what it does. I report three results. First, holding a definition-free round fixed, adding written axis definitions moves scores by a mean of 1.8 points on a 21-point scale and tightens agreement between raters (mean absolute gap 2.81 to 2.50; r 0.81 to 0.89); they make two independent raters agree more closely, which an arbitrary steer would not. Second, across nine models from eight laboratories in two countries, Krippendorff's alpha is 0.86 on both an interval and an ordinal metric, and it stayed put as the panel grew from five raters to nine. That is reliability, the reproducibility of a reading, and not validity, its correctness. Third, where the panel does disagree, the disagreement is informative: the sharpest split, a full-scale divergence on an actor's stance toward its state's foundational order, points to a referent problem, and a blind triple-coding puts about two-thirds of it down to interpretation rather than error. I try to be plain about what the method can't do, including the human validation it still lacks, and I release the instrument and data in full. The worked example is the Middle East and North Africa, but I'd expect the method to carry to any region these standard tools leave out.

Related papers

• A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction [45.496856835431885]
  3D reconstruction models emit per-pixel confidence that downstream systems read as a reliability signal.<n>We score the confidence on four properties, how well it ranks error, whether its level is right on average, whether it holds across the confidence range, and whether its intervals cover the truth.<n>The confidence ranks error well, but the predicted uncertainty is too low when it is read under conditions that are not exactly those of training.
  arXiv  Detail & Related papers  (2026-08-30T10:21:34Z)
• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards [0.0]
  We show that preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail.<n>On the 2,885 MultiPref items where both pools are internally unanimous, so no tie-breaking convention is consulted at all.<n>Yet on both corpora the resulting model leaderboards are bit-identical: Kendall tau = 1.00 with zero of six models displaced.
  arXiv  Detail & Related papers  (2026-08-17T00:19:52Z)
• Split the Labor: Separating Evidence Interpretation from Decision Aggregation [0.0]
  Systems that ask a language model to reach a conclusion from many sources usually conflate them into one prompt.<n>We propose a four-field evidence (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves.<n>The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models.
  arXiv  Detail & Related papers  (2026-08-14T17:24:55Z)
• Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict? [1.8620637029128544]
  benchmark score means nothing without knowing what a trivial method achieves and what the best possible method could achieve.<n>We construct both bounds for a task with a rare kind of ground truth: predicting which sentences a crowd of readers -- highlighting for their own purposes, unpaid, uninstructed, and blind to each other -- marked in 120 web documents.
  arXiv  Detail & Related papers  (2026-08-03T05:09:49Z)
• Language Models Agree With Each Other, Not With Readers [1.8620637029128544]
  We measure convergence against a human reference nobody built for the purpose.<n>2,523 reader mark sets across 120 web documents.<n>No model agrees with readers detectably more than a reader does.
  arXiv  Detail & Related papers  (2026-07-31T10:44:10Z)
• AgentFairBench: Do LLM Agents Discriminate When They Act? [2.3004655342211078]
  AgentFairBench is a cheap, reproducible, multi-domain benchmark for demographic disparity in the actions of LLM agents.<n>It spans three regulator-anchored domains: hiring, lending, and medical triage.<n>A NumPy-only harness computes counterfactual flip rate, mean absolute score difference (MASD), action-rate disparity, and tool-invocation disparity.
  arXiv  Detail & Related papers  (2026-06-15T13:50:26Z)
• Factions Within, Uncertain Across: Within-Document Reader Sub-Groups in Social Highlighting [1.8620637029128544]
  We ask the group-level question on a co-readership platform using a margin-preserving curveball null.<n>Experiment 1 shows readers form strong sub-groups far beyond what shared salience, mark density, and sentence popularity predict.<n>Experiment 2 is that grouping a stable reader trait? Here we are honest about power.
  arXiv  Detail & Related papers  (2026-06-10T03:25:36Z)
• Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle [0.0]
  Reference-free faithfulness metrics verify each atomic claim a model makes against ground truth.<n>We show they share a blind spot: they measure only precision, are the stated claims supported?<n>We make this measurable using Formula 1 telemetry, a domain where strategic ground truth is derived deterministically.
  arXiv  Detail & Related papers  (2026-06-08T11:56:25Z)
• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels [0.0]
  LLM-as-a-judge panels aggregate votes from multiple models.<n>We quantify how far their reliability falls short of the independent-voting ideal.
  arXiv  Detail & Related papers  (2026-05-28T11:48:17Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Interpreting Object-level Foundation Models via Visual Precision Search [54.575247537324344]
  We propose a Visual Precision Search method that generates accurate attribution maps with fewer regions.<n>We show our approach enhances object-level task interpretability over SOTA for Grounding DINO and Florence-2 across various evaluation metrics.<n>Our method can interpret failures in visual grounding and object detection tasks, surpassing existing methods across multiple evaluation metrics.
  arXiv  Detail & Related papers  (2024-11-25T08:54:54Z)
• NLP-CIC @ DIACR-Ita: POS and Neighbor Based Distributional Models for Lexical Semantic Change in Diachronic Italian Corpora [62.997667081978825]
  We present our systems and findings on unsupervised lexical semantic change for the Italian language. The task is to determine whether a target word has evolved its meaning with time, only relying on raw-text from two time-specific datasets. We propose two models representing the target words across the periods to predict the changing words using threshold and voting schemes.
  arXiv  Detail & Related papers  (2020-11-07T11:27:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.