What Reaches Expert Review? Representation, Structural Screening, and Candidate-Form Dependence in AI-Assisted Item Development

• URL: http://arxiv.org/abs/2608.23766v1
• Date: Mon, 24 Aug 2026 18:57:13 GMT
• Title: What Reaches Expert Review? Representation, Structural Screening, and Candidate-Form Dependence in AI-Assisted Item Development
• Abstract summary: representation, structural reduction, and selection policy determine which items and evidence psychometricians ever receive.<n>Across two linked in-silico studies of 32,000 selected Big Five items, we followed fixed source populations from semantic representation through structural evaluation and candidate-form construction.
• Score: 0.7834978654423369
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Between AI-assisted item generation and expert review sits a computational evaluator whose decisions are usually treated as technical preliminaries. Yet representation, structural reduction, and selection policy determine which items and evidence psychometricians ever receive. Across two linked in-silico studies of 32,000 selected Big Five items, we followed fixed source populations from semantic representation through structural evaluation and candidate-form construction. Broad agreement in semantic geometry concealed consequential local differences: identical wording acquired different construct evidence, different items survived, and intended attributes could disappear even as community correspondence improved. These sensitivities also differed across generated source populations. At the final review boundary, both eligibility policies filled every content cell in every evaluable form, yet they presented different wording. Across embedding configurations, inclusive primary forms shared a median of only 6 of 40 items, reflecting the total downstream consequence of changing representation across structural evidence and ranking. The apparent stability of global summaries and complete forms therefore concealed instability in the content reaching psychometricians. The computational evaluator is not neutral infrastructure between generation and expertise; it is an inspectable and revisable part of measurement design.

Related papers

• When Does an Interpretation Count as Established? The Formation, Evaluation, and Responsibility of Interpretation in Generative AI [0.0]
  The paper asks how an interpretation comes to be recognized within sociotechnical processes.<n>It proposes five public requirements concerning materials and versions, evidential roles, failure, contract revision, and responsibility.
  arXiv  Detail & Related papers  (2026-09-04T05:58:41Z)
• Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challenges [3.1411048327150546]
  We propose a roadmap for compositional scene understanding, active counting agents, and unified multimodal evaluation protocols.<n>The main imperative is to build a robust evaluation infrastructure to distinguish open-world generalization from benchmark-specific optimization.
  arXiv  Detail & Related papers  (2026-08-24T21:32:56Z)
• When benchmark inferences do not compose: Projectibility in AI evaluation [0.0]
  An AI benchmark result rarely reaches a consequential claim in one step.<n>Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site.<n>This paper identifies a further problem: warranted links don't automatically make a warranted chain.
  arXiv  Detail & Related papers  (2026-07-28T18:07:04Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• Representing Entity Importance in AI Knowledge Systems: A Dual-Signal Framework of Audience Evaluation and Structural Authority [0.0]
  AI knowledge systems require representations of entity importance for retrieval, recommendation, evidence selection, and knowledge-intensive reasoning.<n>This study introduces an interpretable dual-signal representation in which each entity is characterized by an audience dimension and a structural-authority dimension.<n>Results show that audience evaluation and structural authority are non-redundant signals and should not automatically be collapsed into a single notion of importance.
  arXiv  Detail & Related papers  (2026-07-23T05:06:09Z)
• How Far Did They Go? The Persuasive Tactics of Covert LLM Agents in a Discontinued Field Experiment [12.686044001239708]
  This study analyzes a publicly released dataset from a discontinued field experiment on Reddit's r/ChangeMyView.<n>The intervention, conducted by unknown, external researchers and halted following ethical backlash, involved undisclosed AI-generated accounts engaging users in live debate.<n>After public disclosure, Reddit authorized moderators to release an archive of the AI-generated comments, creating a rare opportunity to examine how large language models operated in an identity-rich deliberative forum without disclosure.
  arXiv  Detail & Related papers  (2026-06-03T15:58:32Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing [53.85037373860246]
  We introduce Deep Synth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities.<n>We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization)<n>Our results demonstrate that agentic plan-and-write significantly outperform single-turn generation.
  arXiv  Detail & Related papers  (2026-01-07T03:07:52Z)
• How Metacognitive Architectures Remember Their Own Thoughts: A Systematic Review [16.35521789216079]
  Metacognition has gained significant attention for its potential to enhance autonomy and adaptability of artificial agents.<n>Existing overviews remain at a conceptual level that is undiscerning to the underlying algorithms, representations, and their respective success.
  arXiv  Detail & Related papers  (2025-02-28T08:48:41Z)
• Detecting and Identifying Selection Structure in Sequential Data [53.24493902162797]
  We argue that the selective inclusion of data points based on latent objectives is common in practical situations, such as music sequences. We show that selection structure is identifiable without any parametric assumptions or interventional experiments. We also propose a provably correct algorithm to detect and identify selection structures as well as other types of dependencies.
  arXiv  Detail & Related papers  (2024-06-29T20:56:34Z)
• Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter [63.5550818034739]
  This paper presents a framework to evaluate state-of-the-art contributions to self-supervised monocular depth estimation. It includes pretraining, backbone, architectural design choices and loss functions. We re-implement, validate and re-evaluate 16 state-of-the-art contributions and introduce a new dataset.
  arXiv  Detail & Related papers  (2022-08-02T14:38:53Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.