One Ruler: A Same-Hands Re-Evaluation of Bivariate Causal Direction on Tuebingen, with a Parameter-Free Compression Baseline

• URL: http://arxiv.org/abs/2606.23767v1
• Date: Mon, 22 Jun 2026 15:43:19 GMT
• Title: One Ruler: A Same-Hands Re-Evaluation of Bivariate Causal Direction on Tuebingen, with a Parameter-Free Compression Baseline
• Abstract summary: We re-evaluate the Tuebingen cause-effect pairs in which every method is run by us on the identical 102 pairs.<n>As a clean reference point we introduce a deliberately minimal baseline.<n>Under the common ruler the ranking differs sharply from the literature.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Headline accuracies on the Tuebingen cause-effect pairs are routinely compared across papers even though each is measured under its authors' own protocol -- different pair subsets, weightings, model-selection, and decision rates. We argue this is the wrong comparison and run the right one: a same-hands re-evaluation in which every method is run by us on the identical 102 pairs, with one strict rule -- no tuning and a decision forced on every pair. As a clean reference point we introduce a deliberately minimal baseline: sorted-conditional compression, which feeds quantized, sorted, first-differenced data to an off-the-shelf compressor (bz2) and has zero fitted parameters. Under the common ruler the ranking differs sharply from the literature. Our baseline reaches 74.7% weighted accuracy (p = 3.7e-7); on the same 100 pairs that SLOPE is evaluated on it scores 76.0%, a 1.2-point gap below the authors' own forced-decision SLOPE (77.2%) that is well inside noise (McNemar p = 0.39). A faithful re-run of RECI lands at 70.7% -- inside the original authors' reported error bar, not the 77.5% often quoted (which we trace to a mis-copied cell). SLOPE's published 82.4% is a decided-subset figure: scoring the authors' own stored output only on the pairs its significance test chose to answer reproduces 81.7%. Under the common ruler the methods cluster in the low-to-mid 70s and the zero-parameter compressor ties the strongest of them. We document the mechanisms that inflate published figures (test-set model selection, significance-gated abstention) and contribute two further results: compression score magnitude is a model-free confounding flag (p = 2.8e-68), and a pre-registered falsification test fails in an instructive way that bounds the method's theoretical interpretation. Code, pre-registrations, and per-pair outputs are released.

Related papers

• The Model Proposes, the Code Disposes: A Pre-Registered Ablation of a Verifier-and-Acceptance Stage in an LLM-Orchestrated Offensive-Security Agent [0.0]
  We evaluate whether a verifier-and-acceptance stage - a model verifier whose verdicts are enforced by deterministic code - changes what an LLM-driven offensive-security agent reports.<n>We report a 15-run exploratory pilot, a pre-registered 20-run confirmatory ablation, and a pre-registered 2 x 2 factorial study with 40 runs across two deliberately vulnerable lab targets.<n>The results support a narrow conclusion: the verifier changes what the system ships, while deterministic code supplies enforcement and auditability.
  arXiv  Detail & Related papers  (2026-09-14T17:13:05Z)
• One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs [38.07475237991597]
  Fairness benchmarks such as BBQ have become the de facto standard for fairness evaluation across major model families.<n>We argue that these benchmarks are too easy to support their role.<n>We call for evaluation suites that cover a broader spectrum of fairness alignment.
  arXiv  Detail & Related papers  (2026-09-14T00:21:26Z)
• Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking [0.5735035463793009]
  Open-ended Theory-of-Mind (ToM) trackers emit valid beliefs absent from finite references.<n>A finite-reference-plus-matcher pipeline marks unmatched outputs false, creating proxy labels that can reverse proper-score model selection.
  arXiv  Detail & Related papers  (2026-08-26T11:36:31Z)
• Explanation Multiplicity: Circuit-Level Interpretability Evidence Does Not Survive Defensible Analytic Variation [0.0]
  EU AI Act requires providers of high-risk systems to file technical documentation describing how the system reaches its decisions.<n>Mechanistic interpretability is the obvious source of such evidence, and circuit discovery is its most developed instrument.<n>We ask whether that evidence survives the condition under which it would be relied upon: two competent analysts, the same system, the same tool, different defensible settings.
  arXiv  Detail & Related papers  (2026-08-13T20:27:56Z)
• The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom [0.0]
  LeModel trains a latent world model with a prediction loss and a single anticollapse regulariser.<n>We reproduce that by independent re-implementation on roughly $25 of rented compute, with all evaluation on one laptop CPU.
  arXiv  Detail & Related papers  (2026-08-10T19:00:51Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why [34.429649156970015]
  A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling.<n>Most of the reported numbers are redundant: Pearson's $r$, Spearman's $$, Kendall's $_b$, the phi coefficient $$, and the Matthews Correlation Coefficient all reduce to a single number.
  arXiv  Detail & Related papers  (2026-05-25T07:31:44Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark [1.9537983097153042]
  Behaviour-Driven Development suites accumulate step-duplicate duplication with documented maintenance cost.<n>We release the largest cross-organisational BDD step corpus to date.
  arXiv  Detail & Related papers  (2026-04-22T11:44:05Z)
• Relational Preference Encoding in Looped Transformer Internal States [0.0]
  We investigate how looped transformers encode human preference in their internal iteration states.<n>Using Ouro-2.6B-Thinking, a 2.6B- parameter looped transformer with iterative refinement, we extract hidden states from each loop iteration.<n>We train lightweight evaluator heads to predict human preference on the Anthropic HH-RLHF dataset.<n>Our pairwise evaluator achieves 95.2% test accuracy on 8,552 unseen examples, surpassing a full-batch L-BFGS probe (84.5%) while the base model remains completely frozen.
  arXiv  Detail & Related papers  (2026-04-10T20:00:49Z)
• Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation [0.0]
  Recent work on chain-of-thought faithfulness reports single aggregate numbers.<n>This paper provides evidence that faithfulness is not an objective, measurable property of a model.
  arXiv  Detail & Related papers  (2026-03-20T17:48:43Z)
• From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation [43.148402136307716]
  Cross-model disagreement among heterogeneous ASR systems can act as a reference-free uncertainty signal.<n>We transcribed 50 publicly available medical audio clips with eight ASR systems spanning commercial APIs and open-source engines.<n>Low-agreement regions were enriched for content disagreements, with the content fraction increasing from 53.9% to 73.9% across quintiles of high-risk mass.
  arXiv  Detail & Related papers  (2026-03-02T13:02:13Z)
• Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models [55.94503936470247]
  Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including judges.<n>Most classical methods assume annotators are conditionally independent given the true label $Yin0,1$, an assumption often violated by LLM judges.<n>We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors.
  arXiv  Detail & Related papers  (2026-01-29T21:26:50Z)
• Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures [87.75098311090642]
  Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed.<n>We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres.
  arXiv  Detail & Related papers  (2025-10-16T12:23:13Z)
• End-to-End Page-Level Assessment of Handwritten Text Recognition [69.55992406968495]
  HTR systems increasingly face the end-to-end page-level transcription of a document. Standard metrics do not take into account the inconsistencies that might appear. We propose a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately.
  arXiv  Detail & Related papers  (2023-01-14T15:43:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.