XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression

• URL: http://arxiv.org/abs/2609.02083v1
• Date: Wed, 02 Sep 2026 04:12:44 GMT
• Title: XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression
• Abstract summary: Cross-axis selection identifies a block with low relative-magnitude and angular hidden-state change.<n>Local boundary reconstruction re-fits the adjacent surviving block to match the original two-block output.<n>XMerge uses no task labels or end-to-end fine-tuning.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Removing complete transformer layers preserves a standard serving architecture, but existing depth-compression methods can lose substantial quality, and the loss varies unpredictably across models. We introduce XMerge, a post-training method with two components. Cross-axis selection identifies a block with low relative-magnitude and angular hidden-state change, and local boundary reconstruction re-fits the adjacent surviving block to match the original two-block output. XMerge uses no task labels or end-to-end fine-tuning, and it introduces neither architectural changes nor additional inference-time parameters. Across seven Llama and Qwen backbones (0.5B-8B), five published baselines, and three layer-reduction levels, its advantage over baselines is largest at the most aggressive removal: at k=4 it ranks first on six of seven backbones on CORE (a 22-task aggregate) and, separately, on six of seven on MMLU (five of seven on both at once), while avoiding the large perplexity increases of several competing operators. In a task-level bootstrap, the 95% confidence intervals for the three largest CORE margins exclude zero; the remaining margins are consistent with ties. Across the 14 (model, regime) cells it is also the only evaluated operator that never collapses, ranking top-2 in both zero-shot and in-context regimes; on a first calibration probe (one backbone) it is the best-calibrated operator. Ablations show that local reconstruction provides most of the gain, while cross-axis fusion helps when the two selection axes disagree. The additional construction cost is recovered through per-token decode savings after roughly tens of thousands of requests.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• LegoLM: Structured Weight Sharing for Large Language Models [0.0]
  We present LegoLM, a structured weight-sharing compression framework for large language models.<n>LegoLM resolves two failure modes: distributional mismatch and outlier dominance.<n>Across GPT-2 small (124M) and Mistral-7B, LegoLM achieves +0.03% PPL degradation at 4.41X compression on Mistral-7B.
  arXiv  Detail & Related papers  (2026-08-09T11:51:14Z)
• ReBRAC-v2: The Return of the King [9.34711333978267]
  We introduce ReBRAC-v2, which directly trains an exact-likelihood normalizing flow as the RL actor.<n>Rather than tuning this recipe separately for every task, we develop a single shared configuration.<n>Across ten common state-based OGBench categories, ReBRAC-v2 averages 74.8 compared to 52.3 for the next-best aggregate result.
  arXiv  Detail & Related papers  (2026-08-02T12:42:20Z)
• Reactive 3D Motion Planning for a Franka Arm via Star-World Workspace Reshaping [0.0]
  Safety inflation can cause nearby obstacles to overlap, violating the disjoint-obstacle assumptions used by many modulation-based reactive planners.<n>We investigate Star-World workspace reshaping for three-dimensional reactive control of a Franka Emika Panda manipulator.<n>We compare reshaped and unreshaped obstacle representations in six PyBullet scenarios using goal attainment, path-length ratio, and computation time.
  arXiv  Detail & Related papers  (2026-07-27T23:12:32Z)
• DeRes: Decoupling Residual Stability and Adaptivity for Scalable CTR Prediction [0.343054185715673]
  Transformer-based CTR models face a growing bottleneck at the residual connection.<n>Recent attention-based residual variants (AttnRes) address parts of this in language models.<n>We present DeRes, which routes each layer through two parallel paths.
  arXiv  Detail & Related papers  (2026-06-06T05:07:16Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scale [54.70985426016736]
  We present SkillDAG, which models inter-skill relationships as a typed directed graph.<n>Each search returns vector matches, typed-edge neighbors, and conflict signals, and a propose-then-commit protocol lets the agent register execution-backed edges.<n>On ALFWorld and SkillsBench with MiniMax-M2.7, SkillDAG reaches 67.1% success and 27.3% reward.
  arXiv  Detail & Related papers  (2026-06-02T02:45:21Z)
• CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs [0.0]
  We introduce CRMA, a residual adapter whose internal mixing matrix M is doubly-stochastic at every forward pass via Sinkhorn normalization.<n>On Mistral-7B across 5 domains 3 seeds, modular per-task LoRA mediates a backbone loss-relative drift from +42.96% +/- 5.5 (naive fine-tuning) to -0.17% +/- 0.17, with disjoint per-seed ranges, and improves prior-task holdout loss by 1.99% +/- 0.54 over a matched frozen-substrate baseline.
  arXiv  Detail & Related papers  (2026-05-29T21:50:23Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• PlueckerNet: Learn to Register 3D Line Reconstructions [57.20244406275875]
  This paper proposes a neural network based method to solve the problem of Aligning two partially-overlapped 3D line reconstructions in Euclidean space. Experiments on both indoor and outdoor datasets show that the registration (rotation and translation) precision of our method outperforms baselines significantly.
  arXiv  Detail & Related papers  (2020-12-02T11:31:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.