A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents

• URL: http://arxiv.org/abs/2606.29719v1
• Date: Mon, 29 Jun 2026 02:55:01 GMT
• Title: A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents
• Abstract summary: We introduce -- comprising the Multimodal Preference Collapse Index (M PCI), evaluator-indexed coupling matrix, and Jensen-Shannon-rho (JSD) -- apply it across eight conditions.<n>Four conditions show strong coupling (N=76; GPT-4o June, qwenrho-plus N=30, symmetric LR, DeepSeek self-eval)<n>The May-to-June GPT-4o drift is the most informative measurement: a diagnostic instrument detecting its own instability.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measurements of proprietary LLM evaluators can become invalid within weeks -- we document one case and provide the diagnostic framework to detect it. We introduce EPC -- comprising the Multimodal Preference Collapse Index (MPCI), evaluator-indexed coupling matrix, and Jensen-Shannon divergence (JSD) -- and apply it across eight experimental conditions (N=112 main + N=10 ablation = 122 unique repetitions, all reported). Coupling coefficients range from 0.00 to 1.18 across per-condition means (CV approx 0.9, n=8 conditions). Four conditions show strong coupling (N=36; GPT-4o May, GPT-4o-mini, Qwen3.7-plus, DashScope 30r); four collapse to near-zero (N=76; GPT-4o June, qwen-plus N=30, symmetric LR, DeepSeek self-eval). The May-to-June GPT-4o drift -- an N=8 re-replication inverting the study's conclusion -- is the most informative measurement: a diagnostic instrument detecting its own instability demonstrates the fragility it was designed to measure. Self-evaluation (97% zero, JSD=0.003) consistently collapses, though floor effects are possible. Output-format confound analysis finds per-strategy aggregate rho=0.89 but per-instance rho=0.219 (p=0.093); PCI reported as preference-convergence metric. We release EPC with all data. The finding is not any single coupling magnitude but the pattern of version-conditional instability that makes single-snapshot evaluator studies unreliable.

Related papers

• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use [28.57781785937644]
  We introduce PRISMS (Probing Representations In Support of Monitoring and Steering), a closed-loop framework that shares a failure-specific neuron basis between sparse detection and activation.<n>Across six models, PRISMS reduces pooled over-calling rate by 80% while increasing tool-required accuracy by 14.2 percentage points.
  arXiv  Detail & Related papers  (2026-07-31T19:03:44Z)
• KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models [0.0]
  KAISEN is a five-phase audit pipeline covering subgroup stratification, disparity measurement, mechanism diagnostics, post-hoc mitigation, and drift monitoring.<n>All results are synthetic with known ground truth and do not establish clinical validity.
  arXiv  Detail & Related papers  (2026-07-30T17:57:18Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Multimodal Evaluator Preference Collapse: Cross-Modal Contagion in Self-Evolving Agents [1.0152838128195467]
  We show that Evaluator Preference Collapse ( EPC) is dramatically amplified in multimodal settings.<n>We then demonstrate a novel phenomenon we term cross-modal contagion.<n>We identify cross-model evaluator architecture as the primary risk factor for preference drift.
  arXiv  Detail & Related papers  (2026-06-15T13:18:20Z)
• Reliable Extraction of Clinical Follow-Up Instructions: A Hybrid Neural-Symbolic Pipeline [43.0484058393522]
  We test a hybrid neural-symbolic pipeline against direct generation.<n>We evaluate on a 2,000-note synthetic outpatient corpus with action-disjoint splits.
  arXiv  Detail & Related papers  (2026-05-26T05:14:33Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes [4.738949927143789]
  We investigate whether linearly decodable failure signals in hidden states can be leveraged to correct those failures.<n>We find that decodable failure structure supports post-generation reliability estimation even when the fixed linear steering family cannot exploit it for correction.
  arXiv  Detail & Related papers  (2026-05-07T05:58:38Z)
• Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score [14.579552536669217]
  Unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation Code.<n>We present the first study of reliability in multimodal unlearning.<n>We introduce the Unified Quality Score (UQS), a metric with weights derived from each metric's Spearman correlation with the metric distance.
  arXiv  Detail & Related papers  (2026-05-04T04:13:00Z)
• High-Dimensional Gaussian Mean Estimation under Realizable Contamination [43.56842719227285]
  We study mean estimation for a Gaussian distribution with identity covariance in $mathbbRd$ under a missing data scheme termed realizable $$-contamination model.<n>In this model an adversary can choose a function $r(x)$ between 0 and $$ and each sample $x$ goes missing with probability $r(x)$.
  arXiv  Detail & Related papers  (2026-03-17T17:04:18Z)
• Detecting Unobserved Confounders: A Kernelized Regression Approach [46.52607207396279]
  Kernel Regression Confounder Detection (KRCD) is a novel method for detecting unobserved confounding in nonlinear observational data under single-environment conditions.<n>Test statistic whose significant deviation from zero indicates unobserved confounding.<n>Experiments on synthetic benchmarks and the Twins dataset demonstrate that KRCD not only outperforms existing baselines but also achieves superior computational efficiency.
  arXiv  Detail & Related papers  (2026-01-01T04:26:02Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Maximizing the nondemolition nature of a quantum measurement via an adaptive readout protocol [28.661156827387043]
  Quantum error correction (QEC) requires non-invasive measurements for fault tolerant quantum computing.<n>We develop a protocol for a $D-$dimensional system that switches to probing only the $D-1$ remaining subspace after a positive result.<n>This strategy minimizes measurement-induced errors by relying on negative-result measurement results that do not perturb the Hamiltonian.
  arXiv  Detail & Related papers  (2025-11-14T05:52:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.