SMITE: Enhancing Fairness in LLMs through Optimal In-Context Example Selection via Dynamic Validation

• URL: http://arxiv.org/abs/2508.17735v1
• Date: Mon, 25 Aug 2025 07:22:08 GMT
• Title: SMITE: Enhancing Fairness in LLMs through Optimal In-Context Example Selection via Dynamic Validation
• Authors: Garima Chhikara, Kripabandhu Ghosh, Abhijnan Chakraborty,
• Abstract summary: This study introduces a novel approach to enhancing Large Language Models (LLMs) performance and fairness.<n>We propose a dynamic validation set, which evolves alongside the test set, replacing the traditional static validation approach.<n>We show that our proposed techniques significantly improve both predictive accuracy and fairness compared to baseline methods.
• Score: 6.853912853826401
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are widely used for downstream tasks such as tabular classification, where ensuring fairness in their outputs is critical for inclusivity, equal representation, and responsible AI deployment. This study introduces a novel approach to enhancing LLM performance and fairness through the concept of a dynamic validation set, which evolves alongside the test set, replacing the traditional static validation approach. We also propose an iterative algorithm, SMITE, to select optimal in-context examples, with each example set validated against its corresponding dynamic validation set. The in-context set with the lowest total error is used as the final demonstration set. Our experiments across four different LLMs show that our proposed techniques significantly improve both predictive accuracy and fairness compared to baseline methods. To our knowledge, this is the first study to apply dynamic validation in the context of in-context learning for LLMs.

Related papers

• Large Language Models are Demonstration Pre-Selectors for Themselves [57.101804269100185]
  In-context learning (ICL) with large language models (LLMs) delivers strong few-shot performance by choosing few-shot demonstrations from the entire training data.<n>FEw yet Essential Demonstration prE-selectoR is a novel pre-selection framework that identifies a representative subset of demonstrations.<n>FEw yet Essential Demonstration prE-selectoR can reduce training data size by over 20% while maintaining performance.
  arXiv  Detail & Related papers  (2025-06-06T12:29:03Z)
• Test-Time Learning for Large Language Models [33.11605667376906]
  We propose a Test-Time Learning (TTL) paradigm for Large Language Models (LLMs)<n>LLMs dynamically adapts to target domains using only unlabeled test data during testing.<n>We demonstrate through experiments that TLM improves performance by at least 20% compared to original LLMs on domain knowledge adaptation.
  arXiv  Detail & Related papers  (2025-05-27T02:18:59Z)
• Realistic Test-Time Adaptation of Vision-Language Models [23.972884634610413]
  Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance.<n>Previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution.<n>Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework.
  arXiv  Detail & Related papers  (2025-01-07T12:17:25Z)
• Fair In-Context Learning via Latent Concept Variables [17.216196320585922]
  Large language models (LLMs) can inherit social bias and discrimination from their pre-training data. We design data augmentation strategies that reduce correlation between predictive outcomes and sensitive variables helping to promote fairness during latent concept learning.
  arXiv  Detail & Related papers  (2024-11-04T23:10:05Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Aligning Language Models with Demonstrated Feedback [58.834937450242975]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors.<n>We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs)<n>We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• ParaICL: Towards Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing.<n>Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples.<n>We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection [35.924633625147365]
  Large Language Models (LLMs) can adapt to new tasks via in-context learning (ICL) In this work, we investigate an active learning approach for ICL, where there is a limited budget for annotating examples. We propose a model-adaptive optimization-free algorithm, termed AdaICL, which identifies examples that the model is uncertain about.
  arXiv  Detail & Related papers  (2023-10-30T22:03:55Z)
• A Lagrangian Duality Approach to Active Learning [119.36233726867992]
  We consider the batch active learning problem, where only a subset of the training data is labeled. We formulate the learning problem using constrained optimization, where each constraint bounds the performance of the model on labeled samples. We show, via numerical experiments, that our proposed approach performs similarly to or better than state-of-the-art active learning methods.
  arXiv  Detail & Related papers  (2022-02-08T19:18:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.