ProtStructQA: A Denotation Threshold in Protein Structural Reasoning

• URL: http://arxiv.org/abs/2606.00451v1
• Date: Sat, 30 May 2026 00:42:38 GMT
• Title: ProtStructQA: A Denotation Threshold in Protein Structural Reasoning
• Abstract summary: ProtStructQA is an executable benchmark for protein structural question answering.<n>We release 382.2K questions covering confidence, distances, predicted error (PAE), solvent exposure, secondary structure, topology and contacts.<n>We find a capability-dependent denotation threshold between Qwen3-1.7B and Qwen3-4B.
• Score: 2.997268954610233
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Protein-language systems are often evaluated by whether they generate plausible biological text, but a structural question has a sharper semantics: it denotes a measurement in a 3D coordinate system. We introduce ProtStructQA, an executable benchmark for protein structural question answering in which each natural-language question is generated from a hidden typed domain-specific language (DSL) program and the answer is obtained by executing that program on an AlphaFold-predicted structure. ProtStructQA releases 382.2K questions covering confidence, distances, predicted aligned error (PAE), solvent exposure, secondary structure, topology and contacts, and held-out compositions: a 330K active benchmark over 10K proteins from four species, plus a 52.2K hard-negative robustness pool. Without fine-tuning, we evaluate Qwen3 models from 0.6B to 8B under direct prompting, chain-of-thought, grammar-constrained executable voting, executable voting with chain-of-thought, and multi-turn ReAct-style tool use, and replicate the headline finding on Gemma-3-1B and Gemma-3-12B. We find a capability-dependent denotation threshold between Qwen3-1.7B and Qwen3-4B: below it, tool-mediated ReAct dominates because models often fail to produce executable denotations; above it, chain-of-thought flips from mostly harmful to strongly beneficial and becomes the strongest strategy on most splits. Parse-failure and family-level analyses show that the threshold is a transition from unparseable language to executable structural denotation, while grammar and execution remain selectively valuable for PAE and secondary-structure queries. ProtStructQA reframes scientific QA as compilation from language to measurement and provides a diagnostic testbed for when language models can map words to executable 3D structural measurements.

Related papers

• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
  We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
  arXiv  Detail & Related papers  (2026-06-21T16:27:24Z)
• Self-Training Doesn't Flatten Language -- It Restructures It: Surface Markers Amplify While Deep Syntax Dies [4.738949927143789]
  Successive self-training on a language model's own outputs is widely characterized as a process of flattening.<n>We provide evidence that this characterization is incomplete.<n>Across eleven generations of self-training on five models, language is not flattened uniformly -- it is restructured.
  arXiv  Detail & Related papers  (2026-05-20T01:44:47Z)
• Structured Intent as a Protocol-Like Communication Layer: Cross-Model Robustness, Framework Comparison, and the Weak-Model Compensation Effect [0.0]
  This paper investigates how reliably structured intent representations preserve user goals across different AI models, languages, and prompting frameworks.<n>We find that structured prompting substantially reduces cross-language score variance relative to unstructured baselines.<n>In a user study, AI-expanded 5W3H prompts reduce interaction rounds by 60 percent and increase user satisfaction from 3.16 to 4.04.
  arXiv  Detail & Related papers  (2026-03-31T16:20:28Z)
• StructLens: A Structural Lens for Language Models via Maximum Spanning Trees [52.040177523973334]
  StructLens is an analytical framework designed to reveal how internal structures relate holistically.<n>Our findings demonstrate that StructLens yields an inter-layer similarity pattern that is distinctively different from conventional cosine similarity.
  arXiv  Detail & Related papers  (2026-02-10T11:30:32Z)
• Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement [66.51979814832332]
  model formulates procedural graph extraction as a multi-round reasoning process with dedicated structural and logical refinement.<n>Experiments demonstrate that model achieves substantial improvements in both structural correctness and logical consistency over strong baselines.
  arXiv  Detail & Related papers  (2026-01-27T04:00:48Z)
• How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction [26.53848099802812]
  Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored.<n>We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean.<n>We propose a diagnostic framework that evaluates models beyond recovery accuracy, including semantic fidelity, logical validity, robustness, sensitivity, and information density.
  arXiv  Detail & Related papers  (2026-01-13T15:03:38Z)
• BRIDGE: Building Representations In Domain Guided Program Verification [67.36686119518441]
  BRIDGE decomposes verification into three interconnected domains: Code, Specifications, and Proofs.<n>We show that this approach substantially improves both accuracy and efficiency beyond standard error feedback methods.
  arXiv  Detail & Related papers  (2025-11-26T06:39:19Z)
• Test Case Generation from Bug Reports via Large Language Models: A Cognitive Layered Evaluation Framework [10.919459368597295]
  We present a systematic evaluation of Large Language Models (LLMs) reasoning in test case generation.<n>We evaluate StarCoder and GPT-4o on Defects4J, GHRB, and mutated variants that introduce linguistic and semantic challenges.
  arXiv  Detail & Related papers  (2025-10-06T20:47:12Z)
• DISPROTBENCH: A Disorder-Aware, Task-Rich Benchmark for Evaluating Protein Structure Prediction in Realistic Biological Contexts [76.59606029593085]
  DisProtBench is a benchmark for evaluating protein structure prediction models (PSPMs) under structural disorder and complex biological conditions.<n>DisProtBench spans three key axes: data complexity, task diversity, and Interpretability.<n>Results reveal significant variability in model robustness under disorder, with low-confidence regions linked to functional prediction failures.
  arXiv  Detail & Related papers  (2025-06-18T23:58:22Z)
• StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs [78.84060166851805]
  StructTest is a novel benchmark that evaluates large language models (LLMs) on their ability to follow compositional instructions and generate structured outputs.<n> Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets.<n>We demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o.
  arXiv  Detail & Related papers  (2024-12-23T22:08:40Z)
• FoldToken: Learning Protein Language via Vector Quantization and Beyond [56.19308144551836]
  We introduce textbfFoldTokenizer to represent protein sequence-structure as discrete symbols. We refer to the learned symbols as textbfFoldToken, and the sequence of FoldTokens serves as a new protein language.
  arXiv  Detail & Related papers  (2024-02-04T12:18:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.