TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness

• URL: http://arxiv.org/abs/2512.01183v1
• Date: Mon, 01 Dec 2025 01:46:36 GMT
• Title: TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness
• Abstract summary: This work presents a systematic investigation of how text perturbations interact with temperature settings across multiple LLM runs.<n>We propose a comprehensive RAG Perturbation-Temperature Analysis Framework that subjects retrieved documents to three distinct perturbation types across varying temperature settings.<n>Our work yields three key contributions: (1) a diagnostic benchmark for assessing RAG robustness, (2) an analytical framework for quantifying perturbation-temperature interactions, and (3) practical guidelines for model selection and parameter tuning under noisy retrieval conditions.
• Score: 3.796841688047966
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The evaluation of Retrieval-Augmented Generation (RAG) systems typically examines retrieval quality and generation parameters like temperature in isolation, overlooking their interaction. This work presents a systematic investigation of how text perturbations (simulating noisy retrieval) interact with temperature settings across multiple LLM runs. We propose a comprehensive RAG Perturbation-Temperature Analysis Framework that subjects retrieved documents to three distinct perturbation types across varying temperature settings. Through extensive experiments on HotpotQA with both open-source and proprietary LLMs, we demonstrate that performance degradation follows distinct patterns: high-temperature settings consistently amplify vulnerability to perturbations, while certain perturbation types exhibit non-linear sensitivity across the temperature range. Our work yields three key contributions: (1) a diagnostic benchmark for assessing RAG robustness, (2) an analytical framework for quantifying perturbation-temperature interactions, and (3) practical guidelines for model selection and parameter tuning under noisy retrieval conditions.

Related papers

• Characterizing and Mitigating the Effects of Device Temperature on RF Fingerprinting Accuracy [3.065243326333468]
  Radio Frequency Fingerprinting (RFFP) has emerged as a promising approach for device authentication by exploiting hardware-specific impairments.<n>RFFP sensitivity to temperature can significantly alter device signatures and degrade classification performance.<n>We propose a novel temperature-aware RFFP framework that explicitly incorporates device temperature information into the learning process.
  arXiv  Detail & Related papers  (2026-07-27T21:00:24Z)
• Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean [51.56484100374058]
  This paper presents a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T [0.1, 1.0] to control evaluation rigor.<n>Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI.
  arXiv  Detail & Related papers  (2026-04-04T09:04:38Z)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition [72.24180896265192]
  We introduce AgentNoiseBench, a framework for evaluating robustness of agentic models under noisy environments.<n>We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios.<n>We then categorize environmental noise into two primary types: user-noise and tool-noise.<n>Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T20:33:10Z)
• Temperature in SLMs: Impact on Incident Categorization in On-Premises Environments [0.0]
  We investigate whether locally executed SLMs can meet this challenge.<n>Results indicate that temperature has little influence on performance, whereas the number of parameters and GPU capacity are decisive factors.
  arXiv  Detail & Related papers  (2025-11-21T19:37:09Z)
• Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality [53.03186946689658]
  Test-time scaling with verification has shown promise in improving the performance of large language models.<n>The effect of verification manifests through interactions of three quantities: (i) the generator's coverage, (ii) the verifier's region of convergence (ROC), and (iii) the sampling algorithm's sub-optimality.<n>We frame verifiable test-time scaling as a transport problem. This characterizes the interaction of coverage, ROC, and sub-optimality.
  arXiv  Detail & Related papers  (2025-10-21T18:05:42Z)
• Exploring the Impact of Temperature on Large Language Models:Hot or Cold? [9.70280446429164]
  We evaluate the impact of temperature in the range of 0 to 2 on data sets designed to assess six different capabilities.<n>Our findings reveal skill-specific effects of temperature on model performance, highlighting the complexity of optimal temperature selection.<n>We propose a BERT-based temperature selector that takes advantage of these observed effects to identify the optimal temperature for a given prompt.
  arXiv  Detail & Related papers  (2025-06-08T21:36:26Z)
• Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation [52.3707788779464]
  We introduce a novel Jensen-Shannon Divergence driven method to Attribute Response to Context (ARC-JSD)<n>ARC-JSD enables efficient and accurate identification of essential context sentences without additional fine-tuning, gradient-calculation or surrogate modelling.<n> Evaluations on a wide range of RAG benchmarks, such as TyDi QA, Hotpot QA, and Musique, using instruction-tuned LLMs in different scales demonstrate superior accuracy and significant computational efficiency improvements.
  arXiv  Detail & Related papers  (2025-05-22T09:04:03Z)
• Open-set Anomaly Segmentation in Complex Scenarios [88.11076112792992]
  This paper introduces ComsAmy, a benchmark for open-set anomaly segmentation in complex scenarios.<n>ComsAmy encompasses a wide spectrum of adverse weather conditions, dynamic driving environments, and diverse anomaly types.<n>We propose a novel energy-entropy learning (EEL) strategy that integrates the complementary information from energy and entropy.
  arXiv  Detail & Related papers  (2025-04-28T12:00:10Z)
• Optimizing Temperature for Language Models with Multi-Sample Inference [47.14991144052361]
  This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different large language models.<n>We provide a comprehensive analysis of temperature's role in performance optimization, considering variations in model architectures, datasets, task types, model sizes, and predictive accuracy.<n>We propose a novel entropy-based metric for automated temperature optimization, which consistently outperforms fixed-temperature baselines.
  arXiv  Detail & Related papers  (2025-02-07T19:35:25Z)
• Adaptive Decoding via Latent Preference Optimization [55.70602730588745]
  We introduce Adaptive Decoding, a layer added to the model to select the sampling temperature dynamically at inference time. Our method outperforms all fixed decoding temperatures across a range of tasks that require different temperatures.
  arXiv  Detail & Related papers  (2024-11-14T18:31:39Z)
• Dynamically Scaled Temperature in Self-Supervised Contrastive Learning [11.133502139934437]
  We focus on improving the performance of InfoNCE loss in self-supervised learning by proposing a novel cosine similarity dependent temperature scaling function. Experimental evidence shows that the proposed framework outperforms the contrastive loss-based SSL algorithms.
  arXiv  Detail & Related papers  (2023-08-02T13:31:41Z)
• Fine-tune your Classifier: Finding Correlations With Temperature [2.071516130824992]
  We analyze the impact of temperature on classification tasks by describing a dataset as a set of statistics computed on representations. We study the correlation between these extracted statistics and the observed optimal temperatures.
  arXiv  Detail & Related papers  (2022-10-18T09:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.