Measuring Alignment With Reader Highlights Net of Position and Length

• URL: http://arxiv.org/abs/2607.27739v1
• Date: Thu, 30 Jul 2026 06:24:38 GMT
• Title: Measuring Alignment With Reader Highlights Net of Position and Length
• Abstract summary: On 120 web documents (at least 12 independent readers each), a language-model importance ranking keeps 38.4% of crowd-marked sentences against 19.9% of their matched neighbours.<n>A single human reader reaches +0.182 - indistinguishable from GPT-5.4 (+0.002 [-0.081, +0.088]) and below 5.
• Score: 1.8620637029128544
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context compression discards most of a document before a language model reads it, and is normally evaluated by downstream task accuracy - which makes another model the judge of what mattered. Naturalistic social highlighting offers a non-circular reference: many people independently marking passages on the same page. But the obvious metric, the fraction of crowd-marked sentences a compressor keeps, is confounded twice: crowd marks are front-loaded and crowd-marked sentences are longer, so any method favouring early or long sentences scores well regardless of readers. We remove both by matching each marked sentence against unmarked sentences of the same document at equal relative depth and equal within-document length rank, and we calibrate every estimator on synthetic nulls built from position and length alone - a step that matters, since depth-only stratification returns a false positive on 20-36% of nulls containing no effect. On 120 web documents (at least 12 independent readers each), a language-model importance ranking keeps 38.4% of crowd-marked sentences against 19.9% of their matched neighbours: an enrichment of +0.196 [+0.148, +0.239], at p = 0.0005 under an exact randomization test that assumes nothing about clustering, and replicated cross-vendor. Naive truncation, whose keep rule is position, correctly falls to +0.003. To give the number a scale: scored identically, on the same budget, against a crowd label recomputed to exclude them, a single human reader reaches +0.182 - indistinguishable from GPT-5.4 (+0.002 [-0.081, +0.088]) and below Claude Opus 5. Classical methods are not null - Luhn's 1958 heuristic reaches +0.088 - so reader selection is partly recoverable by counting words; conditioning additionally on lexical centrality removes only 0.010, so the agreement is not centrality. We also report that a claim in our own prior work does not reproduce on this corpus.

Related papers

• Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models [0.0]
  Flesch Reading Ease (FRE) and the Flesch-Kincaid Grade Level (FKGL) are widely used readability scores for English computed from the same two document statistics.<n>Under a topic model with an explicit sentence-boundary token, both scores converge almost surely to deterministic functions of the document topic distribution.<n>All score variation is mediated by topical composition rather than any residual readability signal.
  arXiv  Detail & Related papers  (2026-08-24T14:41:06Z)
• When AI Writes, Who Gets Cited? Evidence of Citation Monoculture Across Language Models [45.91367912142692]
  We show that different models may still select the same narrow subset, producing citation monoculture without any single citation being wrong.<n>Eleven models from three vendors choose at most ten papers from uniformly random panels of thirty, with real titles and abstracts but fabricated authors, reassigned years, and hidden venues and citation counts.
  arXiv  Detail & Related papers  (2026-08-03T14:27:46Z)
• Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict? [1.8620637029128544]
  benchmark score means nothing without knowing what a trivial method achieves and what the best possible method could achieve.<n>We construct both bounds for a task with a rare kind of ground truth: predicting which sentences a crowd of readers -- highlighting for their own purposes, unpaid, uninstructed, and blind to each other -- marked in 120 web documents.
  arXiv  Detail & Related papers  (2026-08-03T05:09:49Z)
• Language Models Agree With Each Other, Not With Readers [1.8620637029128544]
  We measure convergence against a human reference nobody built for the purpose.<n>2,523 reader mark sets across 120 web documents.<n>No model agrees with readers detectably more than a reader does.
  arXiv  Detail & Related papers  (2026-07-31T10:44:10Z)
• Beyond Similarity: Grounded Agentic Extraction and Expert-Adjudicated Evaluation of Intertextuality in Classical Chinese Histories [3.694051864521789]
  We recast fine-grained intertextuality extraction as an agentic task in which a large language model reads two text units in full.<n>We validate the approach on an exhaustive comparison of the Analects with the Book of Han, where three domain experts adjudicate a pooled multi-model candidate set into a benchmark of 2,533 intertextual pairs.
  arXiv  Detail & Related papers  (2026-07-30T02:34:23Z)
• Contrastive ESA: Human Evaluation of Multiple Translations at Once [45.918020796451536]
  We introduce Contrastive Error Span.<n>(cESA) a protocol that presents multiple translations of the source input (text, video, audio, image).<n>In cESA, the annotator sees multiple translations of the same document, marks major and minor error spans, and then assigns a score from 0% to 100% on absolute scale.<n>We validate cESA using a large-scale human evaluation of English->Japanese translations of 12 models, demonstrating reductions in annotation time and noise compared to standard pointwise evaluation.
  arXiv  Detail & Related papers  (2026-07-29T09:01:59Z)
• From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models [3.5005774502836995]
  Grievance is one of the warning signs analysts look for when assessing threats of violence.<n>It is increasingly measured at scale from online text, most often with word-level lexicons that score by matching weighted terms.<n>We show that grievance is measured more faithfully by reading the surrounding context.
  arXiv  Detail & Related papers  (2026-07-23T05:56:04Z)
• Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity [43.23157850664282]
  We introduce VOIR DIRE, a multimodal benchmark of 626 culturally paired image-prompt artifacts spanning U.S. and mainland Chinese contexts.<n>Across six MLLMs, the bias decomposes into two failures: a positivity-floor calibration failure and an orientation failure.<n>We recommend reporting alignment against each reference pool separately and treating cross-pool divergence as a judge property.
  arXiv  Detail & Related papers  (2026-06-12T15:53:40Z)
• The Long Tail, Not the Front Page: Cold-Start Prediction of Crowd Highlight Salience [1.8620637029128544]
  A social highlighter's most useful signal -- which passages a crowd of readers marks -- exists only for documents people have already read.<n>Can the aggregate crowd salience of a document be predicted from its text before its marks accumulate?<n>Prior work on this data found that zero-shot language models recover highlight locations worse than a trivial lead (position) baseline.<n>We ask whether a model trained on the highlight corpus can beat that baseline.
  arXiv  Detail & Related papers  (2026-06-10T04:37:23Z)
• Embeddings for Preferences, Not Semantics [42.35314862635723]
  Modern AI is opening the door to collective decision-making in which participants express their views as free-form text.<n>Standard text embeddings measure semantic similarity, whereas distances in facility location problems and fair clustering require what we call textitpreferential similarity<n>We show that synthetic training data designed to break this correlation provably shifts the optimal scorer away from nuisance-dominated cosine and significantly improves preference prediction across 11 online deliberation datasets.
  arXiv  Detail & Related papers  (2026-05-08T18:15:14Z)
• Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation [0.0]
  Recent work on chain-of-thought faithfulness reports single aggregate numbers.<n>This paper provides evidence that faithfulness is not an objective, measurable property of a model.
  arXiv  Detail & Related papers  (2026-03-20T17:48:43Z)
• Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique [36.96848724920411]
  We introduce a text-preserving watermarking framework that embeds invisible Unicode characters into documents.<n>We experimentally observe a failure rate of less than 0.1% when detecting a reply after fine-tuning with 50 marked documents.<n>No spurious reply was recovered in over 18,000 challenges, corresponding to a 100%TPR@0% FPR.
  arXiv  Detail & Related papers  (2025-10-07T08:34:08Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• Automatically measuring speech fluency in people with aphasia: first achievements using read-speech data [55.84746218227712]
  This study aims at assessing the relevance of a signalprocessingalgorithm, initially developed in the field of language acquisition, for the automatic measurement of speech fluency.
  arXiv  Detail & Related papers  (2023-08-09T07:51:40Z)
• Counter-GAP: Counterfactual Bias Evaluation through Gendered Ambiguous Pronouns [53.62845317039185]
  Bias-measuring datasets play a critical role in detecting biased behavior of language models. We propose a novel method to collect diverse, natural, and minimally distant text pairs via counterfactual generation. We show that four pre-trained language models are significantly more inconsistent across different gender groups than within each group.
  arXiv  Detail & Related papers  (2023-02-11T12:11:03Z)
• End-to-End Page-Level Assessment of Handwritten Text Recognition [69.55992406968495]
  HTR systems increasingly face the end-to-end page-level transcription of a document. Standard metrics do not take into account the inconsistencies that might appear. We propose a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately.
  arXiv  Detail & Related papers  (2023-01-14T15:43:07Z)
• Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis [79.18261352971284]
  We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation. We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings. SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no human-annotated training data.
  arXiv  Detail & Related papers  (2022-10-10T22:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.