Rethinking Layer Redundancy in Large Language Models: Calibration Objectives and Search for Depth Pruning

• URL: http://arxiv.org/abs/2604.24938v1
• Date: Mon, 27 Apr 2026 19:25:28 GMT
• Title: Rethinking Layer Redundancy in Large Language Models: Calibration Objectives and Search for Depth Pruning
• Abstract summary: We show that different objectives yield qualitatively different redundant layers, and that perplexity and downstream accuracy rankings do not consistently align.<n>Under a fixed objective, however, search algorithms tend to produce similar solutions.
• Score: 4.422051502416615
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work has focused on importance criteria and search algorithms, often treating layer redundancy as an inherent structural property of pretrained networks. In contrast, we adopt a \emph{functional perspective}, where redundancy is jointly influenced by the model and the evaluation objective, suggesting that a universal ranking may not be sufficient. Through an empirical study across three LLM families, two calibration objectives, and seven search algorithms, we observe that different objectives yield qualitatively different redundant layers, and that perplexity and downstream accuracy rankings do not consistently align. Under a fixed objective, however, search algorithms tend to produce similar solutions. Overall, our results suggest that the calibration objective may play a more influential role than the choice of search algorithm, indicating that further attention to objective design could be beneficial.

Related papers

• Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization [0.6875312133832079]
  We introduce Priority-Constrained Descent (PCD) to explicitly exploit hierarchical objective structures.<n>PCD preserves the direction of primary descent whilst allowing for the minimal distortion necessary to guarantee progress on secondary objectives.<n>We evaluate PCD within structured network compression settings, unstructured sparsity and low-rankness, and across a variety of synthetic experiments.
  arXiv  Detail & Related papers  (2026-06-28T17:31:47Z)
• Interest Entanglement: The Hidden Barrier to Blind Super-Resolution Optimization [33.76674489448376]
  We show how to balance fidelity and perceptual quality in image super-resolution tasks.<n>We propose the Shared-Feature-Representation based Super-Resolution framework (SFR), which decouples the learning process of different optimization objectives.<n>To better leverage shared features, we also proposed the InfoSqueeze module, which filters redundant information through a dimensionality reduction and expansion process.
  arXiv  Detail & Related papers  (2026-06-21T06:21:28Z)
• Objective-Induced Bias and Search Dynamics in Multiobjective Unsupervised Feature Selection [2.2282508171588806]
  We study the choice of evaluation objective, the direction of subset-size regularisation, and the initialisation strategy.<n>We show that objective design is central to effective multiobjective unsupervised feature selection.
  arXiv  Detail & Related papers  (2026-05-20T15:14:00Z)
• WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking [60.35109192765302]
  Information seeking is a core capability that enables autonomous reasoning and decision-making.<n>We propose WebLeaper, a framework for constructing high-coverage IS tasks and generating efficient solution trajectories.<n>Our method consistently achieves improvements in both effectiveness and efficiency over strong baselines.
  arXiv  Detail & Related papers  (2025-10-28T17:51:42Z)
• A High-Dimensional Feature Selection Algorithm Based on Multiobjective Differential Evolution [6.912442653561439]
  Multiobjective feature selection seeks to determine the most discriminative feature subset.<n>The proposed method significantly outperforms several state-of-the-art multiobjective feature selection approaches.
  arXiv  Detail & Related papers  (2025-05-09T02:02:49Z)
• Constrained Auto-Regressive Decoding Constrains Generative Retrieval [71.71161220261655]
  Generative retrieval seeks to replace traditional search index data structures with a single large-scale neural network.<n>In this paper, we examine the inherent limitations of constrained auto-regressive generation from two essential perspectives: constraints and beam search.
  arXiv  Detail & Related papers  (2025-04-14T06:54:49Z)
• Active search and coverage using point-cloud reinforcement learning [50.741409008225766]
  This paper presents an end-to-end deep reinforcement learning solution for target search and coverage. We show that deep hierarchical feature learning works for RL and that by using farthest point sampling (FPS) we can reduce the amount of points. We also show that multi-head attention for point-clouds helps to learn the agent faster but converges to the same outcome.
  arXiv  Detail & Related papers  (2023-12-18T18:16:30Z)
• Enhancing Infrared Small Target Detection Robustness with Bi-Level Adversarial Framework [61.34862133870934]
  We propose a bi-level adversarial framework to promote the robustness of detection in the presence of distinct corruptions. Our scheme remarkably improves 21.96% IOU across a wide array of corruptions and notably promotes 4.97% IOU on the general benchmark.
  arXiv  Detail & Related papers  (2023-09-03T06:35:07Z)
• Class Anchor Margin Loss for Content-Based Image Retrieval [97.81742911657497]
  We propose a novel repeller-attractor loss that falls in the metric learning paradigm, yet directly optimize for the L2 metric without the need of generating pairs. We evaluate the proposed objective in the context of few-shot and full-set training on the CBIR task, by using both convolutional and transformer architectures.
  arXiv  Detail & Related papers  (2023-06-01T12:53:10Z)
• Bi-objective Ranking and Selection Using Stochastic Kriging [0.0]
  We consider bi-objective ranking and selection problems in which the two objective outcomes have been observed with uncertainty. We propose a novel Bayesian bi-objective ranking and selection method that sequentially allocates extra samples to competitive solutions. Experimental results show that the proposed method outperforms the standard allocation method, as well as a well-known state-of-the-art algorithm.
  arXiv  Detail & Related papers  (2022-09-05T23:51:07Z)
• Alleviating Search Bias in Bayesian Evolutionary Optimization with Many Heterogeneous Objectives [9.139734850798124]
  We deal with multi-objective optimization problems with heterogeneous objectives (HE-MOPs) A new acquisition function that mitigates search bias towards the fast objectives is suggested. We demonstrate the effectiveness of the proposed algorithm by testing it on widely used multi-/many-objective benchmark problems.
  arXiv  Detail & Related papers  (2022-08-25T17:07:40Z)
• An Empirical Study on Learning and Improving the Search Objective for Unsupervised Paraphrasing [0.0]
  We study the problem of smoothing the noise in the search objective by learning to model the search dynamics. Then, the learned model is combined with the original objective function to guide the search in a bootstrapping fashion. Experimental results show that the learned models combined with the original search objective can indeed provide a smoothing effect.
  arXiv  Detail & Related papers  (2022-03-23T00:30:28Z)
• AP-Loss for Accurate One-Stage Object Detection [49.13608882885456]
  One-stage object detectors are trained by optimizing classification-loss and localization-loss simultaneously. The former suffers much from extreme foreground-background imbalance due to the large number of anchors. This paper proposes a novel framework to replace the classification task in one-stage detectors with a ranking task.
  arXiv  Detail & Related papers  (2020-08-17T13:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.