LLM Sparsity Prior for Robust Feature Selection

• URL: http://arxiv.org/abs/2605.23102v1
• Date: Thu, 21 May 2026 23:34:04 GMT
• Title: LLM Sparsity Prior for Robust Feature Selection
• Abstract summary: We introduce a framework for quantifying the quality of LLM-generated weights.<n>We then propose the LLM Sparsity Prior (LSP), which integrates LLM-generated weights into the prior inclusion probabilities.<n>LSP improves prediction accuracy and identifies clinically relevant features missed by the baselines.
• Score: 3.6910249175649983
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) offer a scalable mechanism to elicit domain-informed prior information for high-dimensional variable selection. However, existing methods such as LLM-Lasso are sensitive to weight quality, with performance degrading substantially when LLM-generated weights are inaccurate. To address this challenge, we first introduce a framework for quantifying the quality of LLM-generated weights, enabling rigorous evaluation of LLM-informed methods across varying weight regimes. We then propose the LLM Sparsity Prior (LSP), which integrates LLM-generated weights into the prior inclusion probabilities of Spike-and-Slab and Spike-and-Slab Lasso models via two interpretable hyperparameters governing global sparsity and weight concentration. Hierarchical hyperpriors on these parameters allow the model to dynamically discount uninformative or misleading weights, improving robustness without sacrificing gains when weights are accurate. Finally, we develop principled prompt engineering strategies and validate the method on a private medical dataset studying Acute Kidney Injury. LSP improves prediction accuracy and identifies clinically relevant features missed by the baselines, with robustness to prompt variation and particular effectiveness in low-data regimes.

Related papers

• An Iterative LLM Framework for SIBT utilizing RAG-based Adaptive Weight Optimization [11.168299220031662]
  This study proposes an adaptive weight optimization framework for SIBT planning, driven by large language models (LLMs)<n>A clinical knowledge base, constructed and queried via retrieval-augmented generation (RAG), enhances the model's domain-specific reasoning.<n>The proposed method was validated on 23 patient cases, showing that the LLM-assisted approach produces plans that are comparable to or exceeding clinically approved and fixed-weight plans.
  arXiv  Detail & Related papers  (2025-09-10T08:54:16Z)
• MSL: Not All Tokens Are What You Need for Tuning LLM as a Recommender [24.03860153639828]
  We propose a novel Masked Softmax Loss (MSL) tailored for fine-tuning large language models (LLMs) on recommendation.<n>MSL improves LML by identifying and masking invalid tokens that could lead to fictitious item descriptions during loss computation.<n>Extensive experiments conducted on four public datasets further validate the effectiveness of MSL, achieving an average improvement of 42.24% in NDCG@10.
  arXiv  Detail & Related papers  (2025-04-05T13:48:33Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• Densing Law of LLMs [81.06644243978101]
  Large Language Models (LLMs) have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases.<n>This paper introduces the concept of textitcapacity density'' as a new metric to evaluate the quality of the LLMs across different scales.
  arXiv  Detail & Related papers  (2024-12-05T16:31:13Z)
• A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs [74.35290684163718]
  A primary challenge in large language model (LLM) development is their onerous pre-training cost. This paper explores a promising paradigm to improve LLM pre-training efficiency and quality by leveraging a small language model (SLM)
  arXiv  Detail & Related papers  (2024-10-24T14:31:52Z)
• GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs [51.02233412547456]
  We introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW) Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget.
  arXiv  Detail & Related papers  (2024-08-27T14:41:14Z)
• STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs [28.70239743254508]
  We present the first structural binarization method for LLM compression to less than 1-bit precision. We observe that some weights in binarized LLMs can be randomly flipped without significant performance degradation. Our approach performs better than other compressed binarization methods while significantly reducing memory requirements.
  arXiv  Detail & Related papers  (2024-08-03T15:07:44Z)
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners [51.32182730502002]
  We introduce Singular-value Diagonal Expansion to refine weight distributions to achieve better quantization alignment.<n>Our plug-and-play weight-quantization methods demonstrate substantial performance improvements over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-07-22T09:45:16Z)
• AdpQ: A Zero-shot Calibration Free Adaptive Post Training Quantization Method for LLMs [22.25748046511075]
  AdpQ is a novel zero-shot adaptive PTQ method for Large Language Models (LLMs) It achieves the state-of-the-art performance in low-precision quantization without requiring any calibration data. Our results achieve the same accuracy as the existing methods on various LLM benchmarks while the quantization time is reduced by at least 10x.
  arXiv  Detail & Related papers  (2024-05-22T05:32:11Z)
• Temporal Scaling Law for Large Language Models [70.74571133406958]
  We propose the novel concept of Temporal Scaling Law, studying how the test loss of an LLM evolves as the training steps scale up.<n>In contrast to modeling the test loss as a whole in a coarse-grained manner, we break it down and dive into the fine-grained test loss of each token position.<n>We derive the much more precise temporal scaling law by studying the temporal patterns of the parameters in the dynamic hyperbolic-law.
  arXiv  Detail & Related papers  (2024-04-27T05:49:11Z)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs [53.31402059062365]
  BiLLM is a groundbreaking 1-bit post-training quantization scheme tailored for pretrained large language models. It achieves for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families.
  arXiv  Detail & Related papers  (2024-02-06T09:26:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.