Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model

• URL: http://arxiv.org/abs/2608.02689v1
• Date: Mon, 03 Aug 2026 08:54:18 GMT
• Title: Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
• Abstract summary: We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention)<n>After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity.<n>We release code, weights, recipes, and the full audit trail, and distill the engineering lessons.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention) linear-attention layers on a single consumer-grade GPU budget, and ask a simple question: what exactly does the conversion break? After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity, yet multiple-choice accuracy stays near random chance (25-29% vs. the teacher's 50.6% on C-Eval). Using a four-permutation diagnostic that rotates answer options while holding content fixed, we show the model sticks to option labels (predicting "A" 81% of the time; 106/161 questions keep the same label under all four rotations) rather than following answer content -- an interface injury that standard distillation metrics cannot see. A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-Eval, label-stickiness roughly halved), after which persona SFT and one round of on-policy DPO preserve benchmark scores within noise. We release code, weights, recipes, and the full audit trail, and distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget.

Related papers

• Safe Error Correction for Language Models: Frozen-Base Adjustment with Capability Preservation [0.0]
  We propose a lightweight logit-level correction module atop a frozen language model.<n>CRN v2 corrects 53.3% of base-model errors while showing no degradation on tested capability benchmarks.<n>None of the alternative configurations we tested exceeded the rank-128 logit result.
  arXiv  Detail & Related papers  (2026-09-14T18:00:15Z)
• Option-Aware Retrieval and Task-Specific VLM Adaptation for Medical VQA [0.0611737116137921]
  We describe our submission to the MedReason 2026 challenge, covering multiple-choice (MCQ) and open-ended (OE) medical visual question answering (VQA)<n>Our first finding is that MCQ retrieval must compare answer emphsemantics rather than answer labels.<n>Our second finding attributes the submitted system's accuracy: holding the task-specific MCQ Low-Rank Adaptation (LoRA) adapter fixed and varying the number (k) of in-prompt retrieved examples changes accuracy by at most one case.
  arXiv  Detail & Related papers  (2026-09-14T13:17:42Z)
• Route, Don't Fix: Regime-Dependent Decoding Correction and a Trajectory-Gated Router for Reliable Clinical LLM Answer Selection [1.4358829531570645]
  Large language models (LLMs) are often deemed unsafe for clinical question answering because of their tendency to hallucinate.<n>Inference-time correction uses the model's internal logit signals, but a fixed transformation need not suit every question.<n>We introduce ALTAS, which reads terminal entropy and late-layer linearity from one forward pass to choose per question between greedy decoding and late-layer trajectory correction.
  arXiv  Detail & Related papers  (2026-09-13T22:36:37Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Capability-Stratified Degradation in Ternary Language Models [0.0]
  Ternary language models restrict weights to $-1,0,+1$, approaching the limit of $log 3 approx 1.585$ bits/weight.<n>We convert Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT)<n>The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning.
  arXiv  Detail & Related papers  (2026-08-28T19:22:38Z)
• WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training [48.290951447286744]
  WDL-OPD is a mixture-constrained co-training method with two trainable policies.<n>An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL.<n>We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.
  arXiv  Detail & Related papers  (2026-08-10T11:22:53Z)
• OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models [39.15861870749305]
  We propose OPTD, On-Policy Transition Distillation with consistency-guided adaptive compression.<n>It samples partial states from the few-step student's own trajectories, uses a frozen, question-only teacher to identify outcome-aligned future candidates, and orders them by current-state confidence.<n>It consistently improves the quality--efficiency trade-off and attains the strongest overall quality-constrained AUP among the evaluated few-step baselines.
  arXiv  Detail & Related papers  (2026-08-03T23:09:43Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation [2.587194279527956]
  CADENCE is a unified framework with a targeted fix for each failure mode.<n>DRIFT schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories.<n>Six components extend it: (A) COVA, a coverage-adaptive schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-gnostic calibration
  arXiv  Detail & Related papers  (2026-07-18T20:16:26Z)
• Know Before You Fetch: Calibrated Retrieval-Budget Allocation for Retrieval-Augmented Generation [10.60293484295824]
  We formulate adaptive RAG as calibrated retrieval-budget allocation.<n>We calibrate sequence log-probability and prefix-logit uncertainty signals into probabilities of correctness.<n>For sequence log-probability, ECE drops from 0.275 to 0.062 on TriviaQA, 0.643 to 0.009 on NQ, and 0.711 to 0.031 on MS MARCO.
  arXiv  Detail & Related papers  (2026-06-29T08:36:33Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Deterministic Decisions for High-Stakes AI. A Zero-Egress Pipeline with the Deployability of RAG and the Accuracy of Machine Learning [0.0]
  Intervention bias is a previously unquantified failure mode of large-language-model educational advisory agents.<n>We show that supervised policy learning eliminates this bias.<n>We also show an Evaluation Gap: LLM-as-judge scoring is blind to intervention bias, rewarding fluent over-prescription rather than decision quality.
  arXiv  Detail & Related papers  (2026-06-28T08:58:17Z)
• Reliability-Asymmetric Spacecraft Autonomy: Co-Designing a Capable Learned GNC Stack with a Verified, Adaptation-Aware Runtime Shield [2.28438857884398]
  Rule-based autonomy is certifiable but brittle, while learned autonomy is capable but hard to verify.<n>We present AMPLE-GNC, a three-tier guidance, navigation, and control stack.<n>Fine-tuning a pretrained 360M model with grammar-constrained decoding gives a hard output-validity guarantee.<n>We show that a latching safe-hold shield can suppress even a capable controller.
  arXiv  Detail & Related papers  (2026-06-24T03:55:21Z)
• From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks [41.99844472131922]
  We provide a benchmark calibration for foundation-model weak labels.<n>The crossover predicted by theory appears at ng100 on PCAM, 20-50 on ISIC, and 250-500 on NIH-CXR.<n>A structured-vs-random noise sign flip on NIH-CXR shows that the rate-only formulation of the bound is incomplete.
  arXiv  Detail & Related papers  (2026-05-23T23:15:07Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification [35.08680804423239]
  Chain-of-thought (CoT) reasoning improves large language models (LLMs) on difficult tasks, but it also makes inference expensive.<n>We propose Latent-Then-Explicit Reasoning (LaTER), a two-stage paradigm that first performs bounded exploration in a continuous latent space.<n>LaTER projects final-layer hidden states back to the input embedding space, preserves the latent KV cache, and uses entropy and model-native stop-token probes to decide when to switch.
  arXiv  Detail & Related papers  (2026-05-08T06:23:58Z)
• When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models [2.064612766965483]
  We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions)<n>Our experiments reveal several concerning findings.
  arXiv  Detail & Related papers  (2026-03-26T23:04:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.