Trustworthiness Calibration Framework for Phishing Email Detection Using Large Language Models

• URL: http://arxiv.org/abs/2511.04728v1
• Date: Thu, 06 Nov 2025 18:14:44 GMT
• Title: Trustworthiness Calibration Framework for Phishing Email Detection Using Large Language Models
• Abstract summary: Large language models (LLMs) such as GPT-4 and LLaMA-3-8B achieve strong accuracy in text classification.<n>This study introduces the Trustworthiness Framework (TCF), a reproducible methodology for evaluating phishing detectors.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Phishing emails continue to pose a persistent challenge to online communication, exploiting human trust and evading automated filters through realistic language and adaptive tactics. While large language models (LLMs) such as GPT-4 and LLaMA-3-8B achieve strong accuracy in text classification, their deployment in security systems requires assessing reliability beyond benchmark performance. To address this, this study introduces the Trustworthiness Calibration Framework (TCF), a reproducible methodology for evaluating phishing detectors across three dimensions: calibration, consistency, and robustness. These components are integrated into a bounded index, the Trustworthiness Calibration Index (TCI), and complemented by the Cross-Dataset Stability (CDS) metric that quantifies stability of trustworthiness across datasets. Experiments conducted on five corpora, such as SecureMail 2025, Phishing Validation 2024, CSDMC2010, Enron-Spam, and Nazario, using DeBERTa-v3-base, LLaMA-3-8B, and GPT-4 demonstrate that GPT-4 achieves the strongest overall trust profile, followed by LLaMA-3-8B and DeBERTa-v3-base. Statistical analysis confirms that reliability varies independently of raw accuracy, underscoring the importance of trust-aware evaluation for real-world deployment. The proposed framework establishes a transparent and reproducible foundation for assessing model dependability in LLM-based phishing detection.

Related papers

• Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction [13.176724997257802]
  Existing document benchmarks are dominated by clean, high-quality samples.<n>We introduce ConfBench, the first calibration-specific benchmark for key information extraction.<n>We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods.
  arXiv  Detail & Related papers  (2026-08-03T07:03:51Z)
• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Reliability Scaling Laws for Quantized Large Language Models [68.39028482878265]
  Quantization is a strategy to build capable and resource-efficient large language models (LLMs) by reducing the bitwidth of the parameters.<n>We first conduct a comprehensive reliability evaluation of quantized LLMs consisting of three key components.<n>A reliability peak occurs for 4-bit quantized models, indicating that quantizing moderately sized models offers the best reliability-efficiency trade-off.
  arXiv  Detail & Related papers  (2026-07-12T17:39:09Z)
• Confidence-Aware Alignment Makes Reasoning LLMs More Reliable [65.44962502963378]
  We introduce CASPO, a framework that aligns token-level confidence with step-wise logical correctness without training a separate reward model.<n>During inference, we propose Confidence-aware Thought (CaT), which dynamically prunes uncertain reasoning branches with negligible O(V) latency.<n>Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency.
  arXiv  Detail & Related papers  (2026-05-08T07:08:25Z)
• Process Supervision of Confidence Margin for Calibrated LLM Reasoning [52.373121066425455]
  Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability.<n>Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation.<n>We introduce Reinforcement Learning with Confidence Margin (textbfRLCM), a calibration-aware RL framework that jointly optimize correctness and confidence reliability.
  arXiv  Detail & Related papers  (2026-04-25T14:40:13Z)
• Trust in One Round: Confidence Estimation for Large Language Models via Structural Signals [13.89434979851652]
  Large language models (LLMs) are increasingly deployed in domains where errors carry high social, scientific, or safety costs.<n>We present Structural Confidence, a single-pass, model-agnostic framework that enhances output correctness prediction.
  arXiv  Detail & Related papers  (2026-02-01T02:35:59Z)
• Test-Time Adaptation for Tactile-Vision-Language Models [6.918033886034668]
  Tactile-vision-language (TVL) models are increasingly deployed in real-world robotic and multimodal perception tasks.<n>Existing test-time adaptation methods provide filtering in unimodal settings but lack explicit treatment of modality-wise reliability under cross-modal shifts.<n>We propose a reliability-aware framework that estimates per-modality reliability from prediction uncertainty and perturbation-based responses.
  arXiv  Detail & Related papers  (2026-01-31T02:26:01Z)
• Tri-LLM Cooperative Federated Zero-Shot Intrusion Detection with Semantic Disagreement and Trust-Aware Aggregation [5.905949608791961]
  This paper introduces a semantics-driven federated IDS framework that incorporates language-derived semantic supervision into federated optimization.<n>The framework achieves over 80% zero-shot detection accuracy on unseen attack patterns, improving zero-day discrimination by more than 10% compared to similarity-based baselines.
  arXiv  Detail & Related papers  (2026-01-30T16:38:05Z)
• ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability [23.70973331911138]
  We argue that usable reasoning systems must be trustworthy, characterized by three properties: interpretability, faithfulness, and reliability.<n>We propose ReFIne, a new training framework that integrates supervised fine-tuning with GRPO to encourage models to improve interpretability.<n>Our experimental results show that ReFIne models generate clearer and better-structured reasoning traces.
  arXiv  Detail & Related papers  (2025-10-10T07:08:44Z)
• Robust Partial 3D Point Cloud Registration via Confidence Estimation under Global Context [12.216399037814012]
  Partial point cloud registration is essential for autonomous perception and 3D scene understanding.<n>We propose Confidence Estimation under Global Context (CEGC), a unified, confidence-driven framework for robust partial 3D registration.<n>CEGC enables accurate alignment in complex scenes by jointly modeling overlap confidence and correspondence reliability within a shared global context.
  arXiv  Detail & Related papers  (2025-09-29T04:36:55Z)
• Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation [63.49409574310576]
  Large language models (LLMs) exhibit overconfidence, assigning high confidence scores to incorrect predictions.<n>We introduce FineCE, a novel confidence estimation method that delivers accurate, fine-grained confidence scores during text generation.<n>Our code and all baselines used in the paper are available on GitHub.
  arXiv  Detail & Related papers  (2025-08-16T13:29:35Z)
• LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models [51.55869466207234]
  Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting.<n>We introduce LLMEval-3, a framework for dynamic evaluation of LLMs.<n>LLEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run.
  arXiv  Detail & Related papers  (2025-08-07T14:46:30Z)
• SteerConf: Steering LLMs for Confidence Elicitation [11.872504642312705]
  Large Language Models (LLMs) exhibit impressive performance across diverse domains but often suffer from overconfidence.<n>We propose SteerConf, a novel framework that systematically steers LLMs' confidence scores to improve their calibration and reliability.
  arXiv  Detail & Related papers  (2025-03-04T18:40:49Z)
• Fact-Level Confidence Calibration and Self-Correction [64.40105513819272]
  We propose a Fact-Level framework that calibrates confidence to relevance-weighted correctness at the fact level. We also develop Confidence-Guided Fact-level Self-Correction ($textbfConFix$), which uses high-confidence facts within a response as additional knowledge to improve low-confidence ones.
  arXiv  Detail & Related papers  (2024-11-20T14:15:18Z)
• Automated Proof Generation for Rust Code via Self-Evolution [69.25795662658356]
  We introduce SAFE, a framework that overcomes the lack of human-written snippets to enable automated proof generation of Rust code.<n> SAFE re-purposes the large number of synthesized incorrect proofs to train the self-ging capability of the fine-tuned models.<n>We achieve a 52.52% accuracy rate in a benchmark crafted by human experts, a significant leap over GPT-4o's performance of 14.39%.
  arXiv  Detail & Related papers  (2024-10-21T08:15:45Z)
• Towards Stable 3D Object Detection [64.49059005467817]
  Stability Index (SI) is a new metric that can comprehensively evaluate the stability of 3D detectors in terms of confidence, box localization, extent, and heading. To help models improve their stability, we introduce a general and effective training strategy, called Prediction Consistency Learning (PCL) PCL essentially encourages the prediction consistency of the same objects under different timestamps and augmentations, leading to enhanced detection stability.
  arXiv  Detail & Related papers  (2024-07-05T07:17:58Z)
• FederatedTrust: A Solution for Trustworthy Federated Learning [3.202927443898192]
  The rapid expansion of the Internet of Things (IoT) has presented challenges for centralized Machine and Deep Learning (ML/DL) methods. To address concerns regarding data privacy, collaborative and privacy-preserving ML/DL techniques like Federated Learning (FL) have emerged.
  arXiv  Detail & Related papers  (2023-02-20T09:02:24Z)
• Prompting GPT-3 To Be Reliable [117.23966502293796]
  This work decomposes reliability into four facets: generalizability, fairness, calibration, and factuality. We find that GPT-3 outperforms smaller-scale supervised models by large margins on all these facets.
  arXiv  Detail & Related papers  (2022-10-17T14:52:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.