A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup

• URL: http://arxiv.org/abs/2605.11554v1
• Date: Tue, 12 May 2026 05:36:46 GMT
• Title: A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup
• Abstract summary: Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another.<n>We test this requirement in a fixed pretraining-and-probing setup motivated by computationally bounded notions of learned structure.<n>A proxy for total learned structure can fail to track the task-relevant structure that drives OOD performance, even in a controlled setting.
• Score: 0.9805949492148788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another, but such explanations require the proxy to track the structure that matters for the downstream task. We test this requirement in a fixed pretraining-and-probing setup motivated by computationally bounded notions of learned structure, including epiplexity. The core question is whether a proxy ranking of two pretraining datasets must agree with their ranking by OOD probe accuracy. We show that it need not. First, we give a controlled construction in which a formal structure quantity, its operational proxy, and the task-relevant structure for a target family separate. We then instantiate the same mechanism in a synthetic sequence-model experiment: under the primary all-sample evaluation, the OOD accuracy ranking reverses the proxy ranking in two of three seeds, with auxiliary diagnostics and ablations supporting the same interpretation. The counterexample does not reject structure-based explanations in general; it identifies a boundary on strong proxy-based explanations. A proxy for total learned structure can fail to track the task-relevant structure that drives OOD performance, even in a controlled setting.

Related papers

• DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction [20.20663425813761]
  DISTAL is a dual-prior framework for structure-agnostic materials property prediction.<n>It combines self-supervised compositional pretraining with structure-aware knowledge distillation.<n> DISTAL achieves the strongest overall performance among all evaluated feature combinations.
  arXiv  Detail & Related papers  (2026-08-30T14:00:55Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• Reasoning without Gold Standards: A Proxy-Judge Theory of Autoformalization [25.540859911959885]
  Autoformalization (AF) asks a model to translate informal mathematical or logical reasoning into a formally checkable object.<n>We introduce a reference-free proxy-judge framework for AF that replaces gold-standard matching with a vector of per-axis property checks.<n>Across seven formalization backbones on miniF2F, ProofNet, e-SNLI, and ProntoQA, refinement consistently lifts Pass Rate over the single-shot ICL baseline.
  arXiv  Detail & Related papers  (2026-06-08T12:57:09Z)
• Deep Reasoning in General Purpose Agents via Structured Meta-Cognition [58.185853639335896]
  We introduce an inference-time approach for constructing task-specific scaffolds through structured meta-reasoning.<n>We instantiate this approach in a general-purpose agent that distributes complex tasks across more controlled reasoning threads.
  arXiv  Detail & Related papers  (2026-05-12T01:21:37Z)
• The Generalized Turing Test: A Foundation for Comparing Intelligence [0.8929189891250133]
  We introduce a formal framework for comparing the capabilities of arbitrary agents via indistinguishability.<n>For agents A and B, we define the Turing comparator A $geq$ B to hold if B, acting as a distinguisher, cannot reliably distinguish between interactions with A and another instance of B.<n>This yields a dataset- and task-agnostic notion of relative intelligence.
  arXiv  Detail & Related papers  (2026-05-11T17:00:18Z)
• EGL-SCA: Structural Credit Assignment for Co-Evolving Instructions and Tools in Graph Reasoning Agents [35.25814217014108]
  We propose EGL-SCA, a verifier-centric dual-space framework for graph reasoning agents.<n>We show that EGL-SCA achieves a state-of-the-art 92.0% average success rate.
  arXiv  Detail & Related papers  (2026-05-11T11:09:32Z)
• Semantic Entanglement in Vector-Based Retrieval: A Formal Framework and Context-Conditioned Disentanglement Pipeline for Agentic RAG Systems [0.0]
  We formalize semantic entanglement as a model-relative measure of cross-topic overlap in embedding space.<n>We introduce the Semantic Disentanglement Pipeline (SDP), a four-stage preprocessing framework that restructures documents prior to embedding.<n>We evaluate SDP on a real-world enterprise healthcare knowledge base comprising over 2,000 documents across approximately 25 sub-domains.
  arXiv  Detail & Related papers  (2026-04-20T00:24:34Z)
• Reasoning Provenance for Autonomous AI Agents: Structured Behavioral Analytics Beyond State Checkpoints and Execution Traces [0.0]
  Agent Execution Record (AER) is a structured reasoning primitive that captures intent, observation, and inference as first-class queryable fields on every step.<n>We show how AERs enable population-level behavioral analytics: reasoning pattern mining, confidence calibration, cross-agent comparison, counterfactual regression testing via mock replay.
  arXiv  Detail & Related papers  (2026-03-23T08:27:54Z)
• Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures [58.54426802984356]
  We introduce a causal evaluation protocol that makes explicit intermediate structures causally determine the output.<n>Across eight models and three benchmarks, models appear self-consistent with their own intermediate structures but fail to update predictions after intervention in up to 60% of cases.<n>Overall, intermediate structures in schema-guided pipelines function as influential context rather than stable causal mediators.
  arXiv  Detail & Related papers  (2026-03-17T13:01:44Z)
• StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs [78.84060166851805]
  StructTest is a novel benchmark that evaluates large language models (LLMs) on their ability to follow compositional instructions and generate structured outputs.<n> Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets.<n>We demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o.
  arXiv  Detail & Related papers  (2024-12-23T22:08:40Z)
• Aligned Unsupervised Pretraining of Object Detectors with Self-training [41.03780087924593]
  Unsupervised pretraining of object detectors has recently become a key component of object detector training. We propose a framework that mitigates this issue and consists of three simple yet key ingredients. We show that our strategy is also capable of pretraining from scratch (including the backbone) and works on complex images like COCO.
  arXiv  Detail & Related papers  (2023-07-28T17:46:00Z)
• Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling [67.75095378830694]
  We develop a two-stage odometry estimation network, where we obtain the ego-motion by estimating a set of sub-region transformations. In this paper, we aim to alleviate the influence of unreliable structures in training, inference and mapping phases. Our two-frame odometry outperforms the previous state of the arts by 16%/12% in terms of translational/rotational errors.
  arXiv  Detail & Related papers  (2022-02-27T12:52:27Z)
• Aligning Pretraining for Detection via Object-Level Contrastive Learning [57.845286545603415]
  Image-level contrastive representation learning has proven to be highly effective as a generic model for transfer learning. We argue that this could be sub-optimal and thus advocate a design principle which encourages alignment between the self-supervised pretext task and the downstream task. Our method, called Selective Object COntrastive learning (SoCo), achieves state-of-the-art results for transfer performance on COCO detection.
  arXiv  Detail & Related papers  (2021-06-04T17:59:52Z)
• Bi-tuning of Pre-trained Representations [79.58542780707441]
  Bi-tuning is a general learning framework to fine-tune both supervised and unsupervised pre-trained representations to downstream tasks. Bi-tuning generalizes the vanilla fine-tuning by integrating two heads upon the backbone of pre-trained representations. Bi-tuning achieves state-of-the-art results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins.
  arXiv  Detail & Related papers  (2020-11-12T03:32:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.