How Much Human Label Variation Does Formal Semantic Structure Explain?: Group-Level Effects and Item-Level Ceilings in NLI

• URL: http://arxiv.org/abs/2607.15870v1
• Date: Fri, 17 Jul 2026 11:34:30 GMT
• Title: How Much Human Label Variation Does Formal Semantic Structure Explain?: Group-Level Effects and Item-Level Ceilings in NLI
• Abstract summary: Human label variation in natural language inference is increasingly treated as signal rather than noise.<n>We measure it on the 3,113 SNLI and MNLI items of ChaosNLI using a rule-based operator and monotonicity tagger validated against MED.<n>In this sample, formal semantic structure shifts how much annotators disagree by a small amount and does not detectably change what they disagree about.
• Score: 1.9177583131035885
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human label variation in natural language inference is increasingly treated as signal rather than noise, but how much of it formal semantic structure explains has not been measured directly. We measure it on the 3,113 SNLI and MNLI items of ChaosNLI, using a rule-based operator and monotonicity tagger validated against MED (0.883 agreement at the edit site, 0.807 on the sentence-level summary our analyses consume), three preregistered analysis blocks, and full reporting of negative results. Three bounds emerge. First, a group-level boundary: hypotheses that are not purely upward monotone show reliably higher label entropy (Cliff's delta = -0.284), and rank-based tests defend the effect against operator-presence and length reductions, though a bounded-outcome sensitivity check weakens the regression form of the length defense. Second, an item-level ceiling: the same formal profiles explain only 3.3 to 3.6 percent of entropy variance and reach a median-split AUC of 0.606, too weak to identify high-disagreement items. Third, composition invariance: across the boundary, three high-powered preregistered contrasts on validated error shares and explanation-type shares (VariErr, LiTEx) all return null results. In this sample, formal semantic structure shifts how much annotators disagree by a small amount and does not detectably change what they disagree about. ChaosNLI-S/M consists of items selected for low original agreement, and every claim is conditioned on that scope. All analyses were preregistered in a version-controlled research log, whose audit trail, including one corrected interpretation rule, the paper discloses.

Related papers

• Off-Target Effects of Response-Style Alignment in a Korean 27B Language Model [2.8678936590409028]
  We measure two behaviours the objective never targets: abstention on ambiguous social questions and unprompted disclosure in securities guidance.<n>Matched target-form controls show that answer propensity depends on the training target, not the prompt set or recipe alone.
  arXiv  Detail & Related papers  (2026-09-10T09:23:04Z)
• Deposon: An Auditable, Conservation-Guaranteed, Game-Theoretically Tested Scattering Layer over LLM Reasoning Paths [0.0]
  Multi-step LLM reasoning lacks a machine-recheckable ledger.<n>We propose the Deposon scattering layer, which binds each node of an LLM-generated concept-decomposition graph.
  arXiv  Detail & Related papers  (2026-09-08T16:40:31Z)
• Boundary-Mutation Testing for Pattern-Based Secret Detection: A Rule-Level Method and Cross-Scanner Evaluation [0.0]
  Boundary-mutation testing for pattern-based secret scanners.<n>Three scanners - a 43-rule open-source scanner, Gitleaks 8.21.2, and TruffleHog 3.82.13 - are tested.
  arXiv  Detail & Related papers  (2026-09-02T13:41:50Z)
• The Imperfective Paradox Is Not Necessarily in Large Language Models: A Benchmark Failure Before a Model Failure [42.25959731143407]
  We examine the benchmark and conclusions and show that it is substantially affected by conceptual and evaluation mis-specifications.<n>Under a strict NLI standard, 76% of Group A instances do not explicitly rule out culmination.<n>We formulate event-semantic NLI as a Multi-step Reasoning Problem and assess both intermediate semantic decisions and final predictions.
  arXiv  Detail & Related papers  (2026-08-25T18:00:18Z)
• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• Located but Not Releasable: Silent Gate Inversion and Bounded Linear Release [0.0]
  We submit the complete pipeline -- detect, localize, and release -- to a fully preregistered stress test on a 25.7M transformer.<n> Interventions at observation-evidence channels of mid layers restore target behavior on otherwise-suppressed worlds.<n>Gating fails out of distribution: a detector calibrated to trigger on zero out-of-distribution calibration worlds triggers on 6.9-7.3% of held-out in-distribution generations and on zero of the 2,400 held-out generations that actually need it.
  arXiv  Detail & Related papers  (2026-08-12T09:04:54Z)
• Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations [1.9177583131035885]
  An earlier study found that hypotheses containing non-upward monotonicity operators showed lower label agreement.<n>We preregistered a replication of this boundary using the same operator tagger and a four-level ordinal agreement outcome.<n>We conclude that the earlier negative boundary is plausibly a structure conditional on low-agreement selection rather than a population-level property.
  arXiv  Detail & Related papers  (2026-07-21T16:03:05Z)
• The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol [0.0]
  Studies of bias in LLM-as-judge systems typically build synthetic corpora by prompting an LLM to generate a hallucinated answer to pair with a factual one, then presenting both to a judge.<n>We report a case in which this generation step silently failed, and use it to argue that the failure mode is structural rather than incidental.<n>A positive control supports this claim directly: an analogous fault injected into a minimal perturbation-based corpus is caught with 100% accuracy by a zero-cost, zero-human gold-to-negative string comparison.
  arXiv  Detail & Related papers  (2026-07-15T11:12:54Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP) [1.827510863075184]
  Grounding-Aware Sensitivity by Perturbation (GASP) is a span-level detector that scores each answer sentence by how strongly its likelihood depends on the retrieved evidence.<n>GASP holds the answer fixed and re-scores it under the full context, under no context, and with each chunk removed, then measures the log-likelihood drops and Jensen-Shannon divergences.<n>We evaluate GASP on three benchmarks (RAGTruth, TofuEval, RAGBench) with three instruction-tuned scorers from two model families.
  arXiv  Detail & Related papers  (2026-07-05T10:35:30Z)
• AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability [16.27940179594792]
  AdversaBench is an end-to-end red-teaming pipeline that mutates seed prompts with five structured operators.<n>We report experiments on 45 seeds across three categories: reasoning, instruction-following, and tool use.
  arXiv  Detail & Related papers  (2026-06-23T13:50:51Z)
• Measuring the Symmetry--Data Exchange Rate [0.2538209532048867]
  We show that a wrong-group control with identical orbit size and matched compute is worse than no constraint.<n>Relative exchange rate beta_diff = 1.28 is consistent in sign and order of magnitude with the theoretical bootstrap 1.0.<n>The methodological contributions -- the relative-rate estimator that cancels the shared-difficulty confound, the wrong-group control, and a pre-specified failure taxonomy -- transfer to any inductive bias whose strength can be parameterised.
  arXiv  Detail & Related papers  (2026-05-31T08:17:40Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• Pairwise matrices for sparse autoencoders: single-feature inspection mislabels causal axes [2.741152471987327]
  Standard sparse-autoencoder protocol labels each feature from its top-activating contexts and validates by single-feature steering.<n>We propose the pairwise matrix protocol, co-varying steering coefficient with joint condition, and report three findings the standard one-corner protocol misses on Qwen3-1.7B-Instruct, replicated on Gemma-2-2B-it.<n>All three findings reproduce on Gemma with model-specific damage signatures; the matched-geometry control is CI-separated by 10x.
  arXiv  Detail & Related papers  (2026-05-04T21:11:21Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics [70.52570641514146]
  We present a benchmark of unfaithful minimal pairs (BUMP) BUMP is a dataset of 889 human-written, minimally different summary pairs. Unlike non-pair-based datasets, BUMP can be used to measure the consistency of metrics.
  arXiv  Detail & Related papers  (2022-12-20T02:17:30Z)
• Unsupervised Full Constituency Parsing with Neighboring Distribution Divergence [48.69930912510414]
  We propose an unsupervised and training-free labeling procedure by exploiting the property of a recently introduced metric. For implementation, we develop NDD into Dual POS-NDD and build "molds" to detect constituents and their labels in sentences. We show that DP-NDD not only labels constituents precisely but also inducts more accurate unlabeled constituency trees than all previous unsupervised methods with simpler rules.
  arXiv  Detail & Related papers  (2021-10-29T17:27:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.