Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score

• URL: http://arxiv.org/abs/2605.02206v2
• Date: Fri, 08 May 2026 07:12:49 GMT
• Title: Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score
• Abstract summary: Unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation Code.<n>We present the first study of reliability in multimodal unlearning.<n>We introduce the Unified Quality Score (UQS), a metric with weights derived from each metric's Spearman correlation with the metric distance.
• Score: 14.579552536669217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation (GDPR), yet current evaluation practices are inconsistent. We present the first systematic study of metric reliability in multimodal unlearning. Five standard metrics, Forget Accuracy (FA), Retain Accuracy (RA), Membership Inference Attack (MIA), Activation Distance (AD), and JS divergence (JS), yield conflicting method rankings across three VQA benchmarks (MLLMU-Bench, UnLOK-VQA, MMUBench). Kendall tau analysis over 36 unlearned LLaVA-1.5-7B models reveals two opposing clusters, {FA, RA, MIA} and {AD, JS}, with tau_FA_AD = -0.26, reproduced on BLIP-2 OPT-2.7B. Agreement is lower in multimodal VQA (average tau = 0.086) than in unimodal classification (average tau = 0.158; difference = 0.072), indicating that dual image-and-text pathways amplify inconsistency. We introduce the Unified Quality Score (UQS), a composite metric with weights derived from each metric's Spearman correlation with the oracle distance d(M_hat, M_star), where M_star is the oracle model retrained only on the retain set. RA shows the strongest reliability (rho = 0.484, p = 0.003), while FA is negatively correlated (rho = -0.418, p = 0.011). UQS yields stable rankings under 100 random weight perturbations (tau = 0.647 +- 0.262). We release the benchmark, 36 checkpoints, and an interactive leaderboard. Code and pre-computed results are available at https://github.com/neurips26/UnifiedUnl.

Related papers

• Not All Relations Are Equal: Relation-Balanced and Calibrated Graph Learning for Provenance-Based Intrusion Detection [66.44527094471619]
  We present RECAL, an unsupervised framework using relation-balanced masked graph learning to better capture rare interaction patterns.<n>On three DARPA E3 datasets, RECAL achieves F1 scores of 99.99%, 99.93%, and 99.99%, outperforming the best baseline on each dataset by 0.88, 0.82, and 0.42 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-09-15T00:31:01Z)
• Leakage-Robust Evaluation and Data-Scale Sensitivity of Attention-Enhanced Multi-Task Learning for Joint Fault Diagnosis and Remaining Useful Life Estimation [0.0]
  Multi-task deep learning models can be affected by how sliding-window sequences are split into training and test sets.<n>We show that naive splitting can inflate classification accuracy from a genuine 20-60 percent to 99.9 percent.<n>We introduce a chunk-based, leakage-audited splitting protocol and evaluate all models using five seeds, one-way ANOVA, and Tukey HSD tests.
  arXiv  Detail & Related papers  (2026-07-17T20:33:36Z)
• When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models [0.0]
  Multi-model LLM systems are used to beat single-model accuracy.<n>We show that their gain is capped by a quantity the field rarely reports.
  arXiv  Detail & Related papers  (2026-06-25T17:06:06Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Beyond Inference-Only Deployment: Comparing Weight-Based Consolidation Against Cascading Compaction [1.6214121783846343]
  We evaluate a nightly consolidation of interaction knowledge into model weights via synthesis, synthesis, and Low-Rank Adaptation (LoRA) fine-tuning on a single consumer.<n>Across ten realistic software development conversations, cascading compaction retain 36.8 +/- 3.0% of knowledge.<n>LoRA consolidates interaction knowledge into model weights via synthesis, synthesis, and Low-Rank Adaptation (LoRA) fine-tuning on a single consumer.
  arXiv  Detail & Related papers  (2026-05-23T16:57:25Z)
• Calibrated Confidence Estimation for Tabular Question Answering [0.0]
  All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA)<n>A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models.<n>A secondary contribution, structure-aware recalibration, improves AUROC by percentage points over standard post-hoc methods.
  arXiv  Detail & Related papers  (2026-04-14T09:16:53Z)
• When Self-Reference Fails to Close: Matrix-Level Dynamics in Large Language Models [0.0]
  Self-referential statements and meta-cognitive prompts are more stable than paradoxical self-reference on key collapse-related metrics.<n>We propose, as a conjecture, that NCTR forces finite-depth transformers toward dynamical regimes where these problems concentrate.
  arXiv  Detail & Related papers  (2026-04-13T23:23:02Z)
• Good Scores, Bad Data: A Metric for Multimodal Coherence [0.0]
  We introduce the Multimodal Coherence Score (MCS), a metric that evaluates fusion quality independent of any downstream model.<n>We evaluate on 1,000 Visual Genome images using DETR, CLIP, and ViLT, and validate on 150 COCO images with no retraining.
  arXiv  Detail & Related papers  (2026-03-26T21:30:34Z)
• DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles [5.647839536820347]
  We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement to produce well-calibrated confidence estimates.<n>DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline.<n>The learned features generalize across benchmarks with near-zero performance degradation.
  arXiv  Detail & Related papers  (2026-03-21T23:24:12Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• From Global to Granular: Revealing IQA Model Performance via Correlation Surface [83.65597122328133]
  We present textbfGranularity-Modulated Correlation (GMC), which provides a structured, fine-grained analysis of IQA performance.<n>GMC includes a textbfDistribution Regulator that regularizes correlations to mitigate biases from non-uniform quality distributions.<n>Experiments on standard benchmarks show that GMC reveals performance characteristics invisible to scalar metrics, offering a more informative and reliable paradigm for analyzing, comparing, and deploying IQA models.
  arXiv  Detail & Related papers  (2026-01-29T13:55:26Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics [70.52570641514146]
  We present a benchmark of unfaithful minimal pairs (BUMP) BUMP is a dataset of 889 human-written, minimally different summary pairs. Unlike non-pair-based datasets, BUMP can be used to measure the consistency of metrics.
  arXiv  Detail & Related papers  (2022-12-20T02:17:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.