Exact Degeneracy Under Balanced k-Shot Sampling:Consequences for Small-Sample Discriminant Analysis on LLM Embeddings

• URL: http://arxiv.org/abs/2609.09860v1
• Date: Wed, 09 Sep 2026 08:13:54 GMT
• Title: Exact Degeneracy Under Balanced k-Shot Sampling:Consequences for Small-Sample Discriminant Analysis on LLM Embeddings
• Abstract summary: We show that balanced k-shot sampling induces an exact, provable degeneracy in a family of small-sample discriminant estimators.<n>We then evaluate the Kernelgression repaired framework on few-shot text classification on frozen embeddings.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Balanced k-shot sampling draws exactly k labeled examples per class. We show that it induces an exact, provable degeneracy in a family of small-sample discriminant estimators. Under balanced sampling, the within-class scatter operator of Kernelized Linear Principal Component Discriminant Analysis (KLPCDA) is not merely rank-deficient but exactly a scaled orthogonal projector. We derive the consequences in closed form: two of KLPCDA's seven variants have every signal eigenvalue exactly equal, so their eigenvector selection criterion is provably indifferent rather than ill-conditioned, and a third has a provably void objective. This follows from the estimators' construction, not any dataset; we confirm it on frozen sentence embeddings and, separately, on residual-stream activations from a decoder-only generative model. An in-formula tie-break repairs the two repairable variants, with recovery gated by class count: the residual subspace constraint costs 5x more on few-class than many-class datasets (p=0.000001). We then evaluate the repaired framework on few-shot text classification on frozen LLM embeddings (n much smaller than d, up to 4096), across four datasets, three embedding sizes, and three trained baselines (SetFit, LoRA, in-context learning). A properly cross-validated logistic-regression probe still beats every KLPCDA variant on three of four datasets, at every embedding size; guidance carried from pixel, vibration-signal, and gene-expression data does not directly generalize to this feature space. Three independent geometric separability metrics fail to explain why one high-dimensional decoder-based embedding model underperforms smaller bidirectional encoders, ruling out anisotropy; the gap is substantially an estimation-efficiency effect, not a permanent ceiling, closing by more than 80% when the support set grows from k<=10 to k=30-50 (p=0.00195, both many-class datasets).

Related papers

• Scaling-Score Conformal Prediction for Multi-Target Regression [0.42970700836450476]
  Multi-target regression requires a model to simultaneously predict several related outputs.<n>We propose the scaling-score conformal method, which is model-agnostic (requires only component-wise absolute residuals)
  arXiv  Detail & Related papers  (2026-09-15T12:23:57Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Geometry-Calibrated Closed-Form Shrinkage for SAR Despeckling [0.7782912896579166]
  Synthetic aperture radar (SAR) despeckling is an inverse-recovery problem in which multiplicative non-Gaussian noise must be suppressed.<n>We revisit a nonlocal sparse estimator that applies a log--Yeo--Johnson transformation, stacks similar patches into groups, codes each group on its own left singular basis, and shrinks the resulting coefficients.
  arXiv  Detail & Related papers  (2026-08-15T04:28:27Z)
• Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations [5.218766876318545]
  Linear probes trained on LLM activations are increasingly proposed as deception-detection metrics, yet report AUROC exceeding 0.96 on clean benchmarks while collapsing under distributional shift.<n>This paper systematically pressure-tests probe-based metrics across the Gemma 3 model family (1B-27B parameters)<n>We test four hypotheses about deception encoding: (1) single linear direction, (2) multi-dimensional subspace, (3) convex conic hull, (4) entropy proxy.
  arXiv  Detail & Related papers  (2026-05-27T04:51:55Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Information-Geometric Decomposition of Generalization Error in Unsupervised Learning [0.0]
  We decompose the Kullback--Leibler generalization error (GE) of unsupervised learning into three non-negative components: model error, data bias, and variance.<n>The decomposition is exact for any e-flat model class.
  arXiv  Detail & Related papers  (2026-04-14T06:23:18Z)
• Ensemble Threshold Calibration for Stable Sensitivity Control [0.0]
  We present an end-to-end framework that achieves exact recall with sub-percent variance over tens of millions of geometry pairs.<n>Our approach consistently hits a recall target within a small error, decreases redundant verifications relative to other calibrations, and runs end-to-end on a single TPU v3 core.
  arXiv  Detail & Related papers  (2025-10-02T15:22:28Z)
• Incorporating Arbitrary Matrix Group Equivariance into KANs [69.30866522377694]
  Kolmogorov-Arnold Networks (KANs) have seen great success in scientific domains.<n>We propose Equivariant Kolmogorov-Arnold Networks (EKAN) to broaden their applicability to more fields.
  arXiv  Detail & Related papers  (2024-10-01T06:34:58Z)
• On the Size and Approximation Error of Distilled Sets [57.61696480305911]
  We take a theoretical view on kernel ridge regression based methods of dataset distillation such as Kernel Inducing Points. We prove that a small set of instances exists in the original input space such that its solution in the RFF space coincides with the solution of the original data. A KRR solution can be generated using this distilled set of instances which gives an approximation towards the KRR solution optimized on the full input data.
  arXiv  Detail & Related papers  (2023-05-23T14:37:43Z)
• Few-Shot Non-Parametric Learning with Deep Latent Variable Model [50.746273235463754]
  We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV) NPC-LV is a learning framework for any dataset with abundant unlabeled data but very few labeled ones. We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime.
  arXiv  Detail & Related papers  (2022-06-23T09:35:03Z)
• Confident Sinkhorn Allocation for Pseudo-Labeling [40.883130133661304]
  Semi-supervised learning is a critical tool in reducing machine learning's dependence on labeled data. This paper studies theoretically the role of uncertainty to pseudo-labeling and proposes Confident Sinkhorn Allocation (CSA) CSA identifies the best pseudo-label allocation via optimal transport to only samples with high confidence scores.
  arXiv  Detail & Related papers  (2022-06-13T02:16:26Z)
• Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity [55.29408396918968]
  We study a family of loss functions named label-distributionally robust (LDR) losses for multi-class classification. Our contributions include both consistency and robustness by establishing top-$k$ consistency of LDR losses for multi-class classification. We propose a new adaptive LDR loss that automatically adapts the individualized temperature parameter to the noise degree of class label of each instance.
  arXiv  Detail & Related papers  (2021-12-30T00:27:30Z)
• Robust Self-Supervised Convolutional Neural Network for Subspace Clustering and Classification [0.10152838128195464]
  This paper proposes the robust formulation of the self-supervised convolutional subspace clustering network ($S2$ConvSCN) In a truly unsupervised training environment, Robust $S2$ConvSCN outperforms its baseline version by a significant amount for both seen and unseen data on four well-known datasets.
  arXiv  Detail & Related papers  (2020-04-03T16:07:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.