PruneShift: A Framework for Evaluating Decision Reliability in Structured Pruning

• URL: http://arxiv.org/abs/2608.29765v1
• Date: Sun, 30 Aug 2026 12:52:16 GMT
• Title: PruneShift: A Framework for Evaluating Decision Reliability in Structured Pruning
• Abstract summary: Structured pruning uses surrogate objectives because direct task evaluation over every feasible mask is too expensive.<n>We introduce PruneShift, an evaluation framework that separates broad predictive fidelity, fidelity near selector outputs, and the quality of the selected pruning decision.
• Score: 1.8023933646971537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Structured pruning uses surrogate objectives because direct task evaluation over every feasible mask is too expensive. Most evaluations report average surrogate error or rank correlation on broadly sampled masks. These summaries do not directly test the mask chosen by the surrogate. We introduce PruneShift, an evaluation framework that separates broad predictive fidelity, fidelity near selector outputs, and the quality of the selected pruning decision. We first prove that Spearman and Kendall agreement can approach one while normalized selection regret remains maximal. We then derive sufficient conditions based on uniform error, selector suboptimality, decision margin, density ratio, and comparison mass. The analysis also yields a finite pool certificate with an explicit excess cost bound. Four studies test different links in this argument. External TextbookQA confirmation is heterogeneous: 7 of 20 simultaneous intervals favor the surrogate-selected mask, 6 favor its fixed comparator, and 7 cross zero. On a fixed Natural Questions pool, strict improvement holds in one of four settings. A controlled QQP experiment supports the proposed coverage mechanism in all 16 prespecified endpoints, although the sufficient bounds are conservative. Finally, a restricted OSSCAR reconstruction study on OPT-125M shows better local than broad fidelity in 68 of 75 primary endpoints. Independent fixed-mask confirmation is inconclusive in 24 of 25 endpoints and favors the comparator in one. These results show why predictive fit, decision reliability, and pruning method quality require separate evidence.

Related papers

• Symmetry-Aware Likelihood-Orbit Aggregation for Selective Left-Right Claim Verification [14.108900142001628]
  We introduce Relation-Orbit, a closed-form contrast that assigns eight normalized likelihoods to query-supporting and counterfactual roles.<n>On VSR and GQA across four frozen vision-language models, Relation-Orbit yields higher mean test coverage at a 10% selective-risk calibration target.
  arXiv  Detail & Related papers  (2026-09-15T11:19:03Z)
• Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA [12.797519888387628]
  We propose root-mean-square Robustness-based Sample Prioritization (RMS-RSP) to perturb hidden states at rationale tokens.<n> RMS-RSP improves robust accuracy and semantic consistency by 1.91 and 2.85 points on average, with the same direction on all five datasets.
  arXiv  Detail & Related papers  (2026-09-09T03:58:59Z)
• Reliable Benchmarking of Artifact Detection in Computational Pathology: A Reproducibility and Uncertainty Analysis [1.8864137667201766]
  Methods: The protocol quantifies four sources of variability - test-set sampling, trainingity, partition composition, and undocumented preprocessing.<n>We apply it to an independent reconstruction of a published diffusion-based artifact detector, evaluated on the original 24-slide partition and against a supervised baseline.<n>Results: The method's central mechanism reproduces: the auxiliary contrastive term improves pooled F1 from 0.673 to 0.688 and replicates under a second seed.
  arXiv  Detail & Related papers  (2026-08-31T14:07:15Z)
• Agreement Before Diversity: Verification-First Complementarity for Heterogeneous Language-Model Coordination [11.215813718564283]
  We decouple candidate headroom from replacement authority, rendering the latter as an explicit, auditable object.<n>Our proposed method, Agreement-Before-Diversity, is a frozen, label-free decision rule.
  arXiv  Detail & Related papers  (2026-08-05T09:24:39Z)
• Cross-Fitted Residual Utility for Primary-Preserving Cognitive Decision Correction in Automatic Modulation Classification [1.547549252134621]
  We study a post-inference problem through cross-fitted residual utility and a primary-preserving cognitive decision policy.<n>A structured KAN-Fourier classifier supplies the default probability, while neural and non-neural candidates provide observable evidence.<n>On RMLA, RMLB, and HISAR, the complete system improves overall accuracy from 63.632% to 66.332%, 65.161% to 66.168%, and 77.769% to 79.867%.
  arXiv  Detail & Related papers  (2026-08-03T11:07:37Z)
• Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration [8.515185888009833]
  Test-time collaboration is often credited with improving LLM reasoning, yet its gains are uneven and sometimes negative.<n>For a fixed candidate pool, we decompose a selector or verifier's net gain into measurable factors.<n>This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology.
  arXiv  Detail & Related papers  (2026-07-20T04:09:19Z)
• Instance-Optimal Estimation with Multiple LLM Judges on a Budget [84.31744861038106]
  We formalize this question as *budgeted heteroskedastic multi-judge estimation*.<n>Given $K$ prompt-response pairs, $J$ judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an $ell_p$-error.<n>We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget.
  arXiv  Detail & Related papers  (2026-05-22T08:26:08Z)
• Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning [8.024041325202612]
  Chain-of-thought (CoT) reasoning with self-consistency improves performance by aggregating multiple sampled reasoning paths.<n>We introduce a conformal procedure for CoT reasoning that directly addresses aggregation uncertainty.<n>Our approach replaces majority voting with weighted score aggregation over reasoning paths and calibrates an abstention rule using conformal risk control.
  arXiv  Detail & Related papers  (2026-05-13T20:33:59Z)
• Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation [47.91529693614168]
  Existing methods are predominantly answer-first, producing confidence only after generating an answer.<n>We study a confidence-first paradigm, where the model outputs its confidence before answering, interpreting this score as the model's probability of answering the question correctly.<n>We propose CoCA, a reinforcement learning framework that jointly optimize confidence calibration and answer accuracy via segmented credit assignment.
  arXiv  Detail & Related papers  (2026-03-06T04:03:13Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)
• Masksembles for Uncertainty Estimation [60.400102501013784]
  Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable.
  arXiv  Detail & Related papers  (2020-12-15T14:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.