Evaluating Nonuniform Dependability Across Response Conditions: A Conditional Generalizability Framework Illustrated in Automated Essay Scoring

• URL: http://arxiv.org/abs/2607.11981v1
• Date: Mon, 13 Jul 2026 08:32:03 GMT
• Title: Evaluating Nonuniform Dependability Across Response Conditions: A Conditional Generalizability Framework Illustrated in Automated Essay Scoring
• Abstract summary: This study introduces a conditional generalizability framework with three components.<n>First, automated scoring configurations are treated as a universe of admissible measurement conditions.<n>Second, analytical D-study projections are compared with empirical configuration sweeps over a finite scoring pool.
• Score: 1.2333280983346018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aggregate reliability estimates can obscure heterogeneity in measurement-design burden across response conditions, so a single G- or D-study may mischaracterize a design's adequacy for particular strata. This study introduces a conditional generalizability framework with three components. First, automated scoring configurations -- the encoder architectures and scoring-head families admissible within a fixed pipeline -- are treated as a universe of admissible measurement conditions rather than incidental modeling choices. Second, analytical D-study projections are compared with empirical configuration sweeps over a finite scoring pool, yielding two estimands of design adequacy whose agreement or divergence diagnoses the realized configuration universe. Third, evidence is conditioned on entropy-defined response strata, treating entropy as an operational stratification variable, not a construct claim about writing quality. Whereas recent generalizability-theory extensions address AI-generated item variants on the response side, this framework addresses the analogous scoring-side problem: AI-mediated scoring configurations. Demonstrated with automated essay scoring of timed L2 writing, the realized design was dependable in aggregate (Phi approx 0.76). Re-estimated within entropy strata, dependability stayed high but declined modestly and robustly (Phi = 0.88, 0.87, 0.84) -- a gradient implying different decision-study requirements, the highest-entropy stratum requiring the most crossed conditions. The framework offers a portable workflow for evaluating nonuniform dependability.

Related papers

• CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement [50.91045324738942]
  CLAIM is an uncertainty-driven framework for active clarification learning in open-domain settings.<n>We propose an entropy-driven synthetic data generation pipeline that integrates uncertainty estimation with semantic clustering and reasoning-based judgments.<n> Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data.
  arXiv  Detail & Related papers  (2026-08-12T04:27:45Z)
• Position: It's Time to Optimize LLMs for Self-Consistency [50.68587056599734]
  Many important failures persist in language model pre- and post-training pipelines.<n>We argue that these failures arise from a modeling assumption permeating all aspects of the pipeline.<n>We propose self-consistency as a framework for understanding these failures.
  arXiv  Detail & Related papers  (2026-07-31T19:16:53Z)
• APO: Unsupervised Atomic Policy Optimization for 3D Structure Prediction of Atomic Systems [64.56252173115246]
  We propose (Atomic Policy Optimization) a fully unsupervised alignment framework that eliminates the need for ground-truth reference structures.<n>APO adapts group-relative policy optimization to 3D atomic environments, utilizing a novel dual-reward mechanism.<n>Our results suggest that intrinsic physical consistency can serve as a superior guide for alignment compared to noisy, supervised coordinate matching.
  arXiv  Detail & Related papers  (2026-07-30T17:21:58Z)
• Gatekeepers and Hallucinations: A Layered Evaluation Framework for LLM-Driven Quantum Circuit Generation [0.0]
  We present a layered evaluation framework for materials-informed Variational Quantum Eigensolver (VQE) circuit generation.<n>We surface a taxonomy of five distinct LLM failure modes (geometry hallucination, nonexistent API usage, integration failures, constraint violations, and plausible-but-unverifiable output)<n>A forensic audit of the evaluation platform's own source code further establishes that two apparent model failures originated in the harness through silent fallback-template substitution.
  arXiv  Detail & Related papers  (2026-06-16T19:17:54Z)
• Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty [5.839104770457067]
  Large language models can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently.<n>We propose structural uncertainty, a consistency-aware framework derived from the stability of self-preference-induced rankings.
  arXiv  Detail & Related papers  (2026-06-15T21:39:47Z)
• R-APS: Compositional Reasoning and In-Context Meta-Learning for Constrained Design via Reflective Adversarial Pareto Search [18.8323743697237]
  Large language models (LLMs) are fluent on open-ended tasks, yet in agentic settings, fluency does not ensure reliable delivery.<n>We trace this gap to three coupled structural failures: errors propagate without localization, worst-case perturbations go unevaluated, and accumulated knowledge is never invalidated.<n>We argue these share a root cause: abductive, counterfactual, meta-inductive, corrective, and inductive reasoning pull a shared context in incompatible directions.
  arXiv  Detail & Related papers  (2026-06-03T12:45:39Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• From Perception to Autonomous Computational Modeling: A Multi-Agent Approach [0.0]
  We present a solver-agnostic framework in which coordinated large language model (LLM) agents autonomously execute the complete computational mechanics workflow.<n>Agents are formalised as conditioned operators on a shared context space with quality gates that introduce conditional iteration between pipeline layers.
  arXiv  Detail & Related papers  (2026-04-08T07:56:34Z)
• Evaluating Robustness of Reasoning Models on Parameterized Logical Problems [20.78623024814435]
  Logic provides a controlled testbed for evaluating LLM-based reasoners.<n>Standard SAT-style benchmarks often conflate surface difficulty (length, wording, clause order) with the structural phenomena that actually determine satisfiability.<n>We introduce a diagnostic benchmark for 2-SAT built from parameterized families of structured 2--CNF formulas.
  arXiv  Detail & Related papers  (2026-02-13T06:54:25Z)
• Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure [58.89643769707751]
  We study latent chain-of-thought as a manipulable causal process in representation space.<n>We find that latent-step budgets behave less like homogeneous extra depth and more like staged functionality with non-local routing.<n>These results motivate mode-conditional and stability-aware analyses as more reliable tools for interpreting and improving latent reasoning systems.
  arXiv  Detail & Related papers  (2026-02-09T15:25:12Z)
• The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving [57.652356955571065]
  State-of-the-art large language model (LLM) pipelines rely on bootstrapped reasoning loops.<n>We analyze how this design choice is sensitive to the collapse of the model's distribution over reasoning paths.<n>We introduce Distributional Creative Reasoning (DCR), a unified variational objective that casts training as gradient flow through probability measures on solution traces.
  arXiv  Detail & Related papers  (2026-01-02T17:10:31Z)
• DISPROTBENCH: A Disorder-Aware, Task-Rich Benchmark for Evaluating Protein Structure Prediction in Realistic Biological Contexts [76.59606029593085]
  DisProtBench is a benchmark for evaluating protein structure prediction models (PSPMs) under structural disorder and complex biological conditions.<n>DisProtBench spans three key axes: data complexity, task diversity, and Interpretability.<n>Results reveal significant variability in model robustness under disorder, with low-confidence regions linked to functional prediction failures.
  arXiv  Detail & Related papers  (2025-06-18T23:58:22Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling [67.75095378830694]
  We develop a two-stage odometry estimation network, where we obtain the ego-motion by estimating a set of sub-region transformations. In this paper, we aim to alleviate the influence of unreliable structures in training, inference and mapping phases. Our two-frame odometry outperforms the previous state of the arts by 16%/12% in terms of translational/rotational errors.
  arXiv  Detail & Related papers  (2022-02-27T12:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.