MACA: A Framework for Distilling Trustworthy LLMs into Efficient Retrievers

• URL: http://arxiv.org/abs/2601.00926v1
• Date: Thu, 01 Jan 2026 23:31:02 GMT
• Title: MACA: A Framework for Distilling Trustworthy LLMs into Efficient Retrievers
• Abstract summary: We present Metadata-Aware Cross-Model Alignment (MACA), which distills a calibrated metadata aware LLM re-ranker into a compact student retriever.<n>A metadata-aware prompt verifies the teacher's trustworthiness by checking consistency under permutations and robustness to paraphrases, then supplies listwise scores, hard negatives, and calibrated relevance margins.
• Score: 7.123910165277138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern enterprise retrieval systems must handle short, underspecified queries such as ``foreign transaction fee refund'' and ``recent check status''. In these cases, semantic nuance and metadata matter but per-query large language model (LLM) re-ranking and manual labeling are costly. We present Metadata-Aware Cross-Model Alignment (MACA), which distills a calibrated metadata aware LLM re-ranker into a compact student retriever, avoiding online LLM calls. A metadata-aware prompt verifies the teacher's trustworthiness by checking consistency under permutations and robustness to paraphrases, then supplies listwise scores, hard negatives, and calibrated relevance margins. The student trains with MACA's MetaFusion objective, which combines a metadata conditioned ranking loss with a cross model margin loss so it learns to push the correct answer above semantically similar candidates with mismatched topic, sub-topic, or entity. On a proprietary consumer banking FAQ corpus and BankFAQs, the MACA teacher surpasses a MAFA baseline at Accuracy@1 by five points on the proprietary set and three points on BankFAQs. MACA students substantially outperform pretrained encoders; e.g., on the proprietary corpus MiniLM Accuracy@1 improves from 0.23 to 0.48, while keeping inference free of LLM calls and supporting retrieval-augmented generation.

Related papers

• AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty [7.105430294335961]
  Multimodal large language models (MLLMs) are widely used for automated annotation, yet their per-class accuracy varies widely.<n>This paper systematically evaluates its full-frame formulation, termed AnchorScore, that assesses the classes MLLMs are least likely to annotate reliably.
  arXiv  Detail & Related papers  (2026-08-17T15:10:33Z)
• Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies [0.9867902290081937]
  Grounded claim factuality checking is important for large language model (LLM) applications such as retrieval-augmented generation.<n>We address this by formulating grounded claim factuality checking as a true/false reading comprehension task.<n>Our method reduces token usage by over 80% compared to unguided open-ended reasoning.
  arXiv  Detail & Related papers  (2026-05-28T10:11:42Z)
• The Erasure Illusion: Stress-Testing the Generalization of LLM Forgetting Evaluation [15.252787015786796]
  Machine unlearning aims to remove specific data influences from trained models.<n>Current unlearning metrics measure success by monitoring the model's performance degradation on a specific unlearning dataset.<n>We argue that for Large Language Models (LLMs), this evaluation paradigm is insufficient and potentially misleading.
  arXiv  Detail & Related papers  (2025-12-22T04:42:41Z)
• Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Aware Unlearning with Proxy Constraint [28.25159814956888]
  Large language models (LLMs) are trained on massive datasets that may include private or copyrighted content.<n>Due to growing privacy and ownership concerns, data owners may request the removal of their data from trained models.<n>Most existing methods lack a sound forgetting boundary, causing some samples to be under-forgotten.
  arXiv  Detail & Related papers  (2025-08-28T05:45:40Z)
• BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap [18.68387394444096]
  Machine unlearning has the potential to improve the safety of large language models (LLMs) by removing sensitive or harmful information post hoc.<n>A key challenge in unlearning involves balancing between forget quality (effectively unlearning undesirable information) and retain quality (maintaining good performance on other, general tasks)<n>We present $textttBLUR$: a benchmark for LLM unlearning that provides more realistic scenarios of forget-retain overlap.
  arXiv  Detail & Related papers  (2025-05-28T22:09:04Z)
• In a Few Words: Comparing Weak Supervision and LLMs for Short Query Intent Classification [4.037445459586932]
  We empirically compare user intent classification into informational, navigational, and transactional categories.<n>Our results indicate that while LLMs outperform weak supervision in recall, they continue to struggle with precision.
  arXiv  Detail & Related papers  (2025-04-30T07:54:04Z)
• Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation [96.18720164390699]
  This paper explores the use of large language models (LLMs) for annotating document utility in training retrieval and retrieval-augmented generation (RAG) systems.<n>Our results show that LLM-generated annotations enhance out-of-domain retrieval performance and improve RAG outcomes compared to models trained solely on human annotations or downstream QA metrics.
  arXiv  Detail & Related papers  (2025-04-07T16:05:52Z)
• Detecting Training Data of Large Language Models via Expectation Maximization [62.28028046993391]
  We introduce EM-MIA, a novel membership inference method that iteratively refines membership scores and prefix scores via an expectation-maximization algorithm.<n> EM-MIA achieves state-of-the-art results on WikiMIA.
  arXiv  Detail & Related papers  (2024-10-10T03:31:16Z)
• LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints [86.59857711385833]
  We introduce RealInstruct, the first benchmark designed to evaluate LLMs' ability to follow real-world multi-constrained instructions. To address the performance gap between open-source and proprietary models, we propose the Decompose, Critique and Refine (DeCRIM) self-correction pipeline. Our results show that DeCRIM improves Mistral's performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
  arXiv  Detail & Related papers  (2024-10-09T01:25:10Z)
• MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators [53.91199933655421]
  Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment.<n>We introduce a universal and training-free framework, $textbfMQM-APE, based on the idea of filtering out non-impactful errors.<n>Experiments show that our approach consistently improves both the reliability and quality of error spans against GEMBA-MQM.
  arXiv  Detail & Related papers  (2024-09-22T06:43:40Z)
• MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector [32.15773300068426]
  Existing methods have designed various sophisticated MIA score functions to achieve considerable detection performance. We propose MIA-Tuner, a novel instruction-based MIA method, which instructs LLMs themselves to serve as a more precise pre-training data detector. We design two instruction-based safeguards to respectively mitigate the privacy risks brought by the existing methods and MIA-Tuner.
  arXiv  Detail & Related papers  (2024-08-16T11:09:56Z)
• Fake Alignment: Are LLMs Really Aligned Well? [91.26543768665778]
  This study investigates the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization.
  arXiv  Detail & Related papers  (2023-11-10T08:01:23Z)
• Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognition [68.68300358332156]
  We propose the sparse conditional hidden Markov model (Sparse-CHMM) to evaluate noisy labeling functions. Sparse-CHMM is optimized through unsupervised learning with a three-stage training pipeline. It achieves a 3.01 average F1 score improvement on five comprehensive datasets.
  arXiv  Detail & Related papers  (2022-05-27T20:47:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.