When Does Span-Guided Detoxification Help? Human Preferences and Evaluator Diagnostics in a Controlled Comparison

• URL: http://arxiv.org/abs/2607.26795v1
• Date: Wed, 29 Jul 2026 11:37:12 GMT
• Title: When Does Span-Guided Detoxification Help? Human Preferences and Evaluator Diagnostics in a Controlled Comparison
• Abstract summary: Span-guided rewriting aims to preserve meaning by localizing edits to annotated harmful spans, but the same constraint can leave harmful intent insufficiently mitigated.<n>We conduct a dense blinded human evaluation under a fixed single-generator setting.<n>Human preferences reveal a trade-off rather than a uniformly superior rewriting strategy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Span-guided rewriting aims to preserve meaning by localizing edits to annotated harmful spans, but the same constraint can leave harmful intent insufficiently mitigated. We present a controlled exploratory comparison of span-guided and unguided detoxification on a mixed-source English evaluation set comprising manually curated inputs and HateXplain test items. We conduct a dense blinded human evaluation under a fixed single-generator setting. Human preferences reveal a trade-off rather than a uniformly superior rewriting strategy. Span-guided outputs are favored when localized editing preserves the original stance and avoids unnecessary modification, whereas unguided outputs are favored when broader rewriting achieves more complete mitigation. This contrast varies substantially across the study-defined strata: the two strategies are competitive in the strong stratum, while unguided rewriting is clearly preferred in the mild stratum. Rationale annotations trace this difference to complementary failure risks: residual harm after localized editing and over-modification after broader rewriting. We treat automatic evaluation as a diagnostic rather than a substitute for human judgment. Toxicity-similarity scalarizations, a multi-generator analysis, and two general-purpose LLM judges reproduce parts of the aggregate tendency but do not yield an analogous stratified contrast. These setting-specific findings do not establish a severity-based routing rule. Instead, they motivate evaluation protocols that assess mitigation sufficiency and meaning preservation separately and report both residual harm and over-modification alongside aggregate scores.

Related papers

• Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores [31.709549159768727]
  We observe a consistent readout gap across diverse reasoning benchmarks.<n> hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses.<n>Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic.
  arXiv  Detail & Related papers  (2026-08-31T16:43:55Z)
• A Controlled Audit of Architectural Complexity in Uncertainty-Aware Multi-Organ Ultrasound Classification [19.598719060464866]
  Multi-organ ultrasound classifiers increasingly combine attention, mixture-of-experts routing, uncertainty gating, and evidential deep learning (EDL) objectives.<n>We contribute a controlled complexity-audit framework, applied to the deployment decision between the maximal evidential candidate Full-EDL and simpler alternatives.
  arXiv  Detail & Related papers  (2026-08-28T08:30:43Z)
• When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide [0.0]
  We benchmark six estimators across five datasets and two known-effect sweeps, and validate the mechanisms against a non-simulated paired reference.<n>Sharpening a logger from the target's own score barely moves overlap over the tested range; action-level disagreement collapses it.<n>Cross-fitting the outcome nuisance alone leaves the reuse bias in place and makes it worse.
  arXiv  Detail & Related papers  (2026-08-12T18:10:10Z)
• LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition [19.651323641444726]
  deterministic TEDS evaluation provides a controlled testbed, using FinTabNet and OmniDocBench.<n>Judge signals were weak on both datasets.<n>severe losses occurred even without specific judge feedback.
  arXiv  Detail & Related papers  (2026-07-15T00:14:31Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement [50.34437999083224]
  Autoraters are increasingly used for evaluation and automated moderation content.<n>There is limited statistical analysis of how modifications in a rubric presented to both humans and autoraters affect their score agreement.
  arXiv  Detail & Related papers  (2026-05-07T13:55:11Z)
• How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation [0.38991526486631006]
  We show that when preference signal is diffuse across prompts, proportional allocation is minimax-optimal.<n>Our results show that inconclusive or negative human evaluation outcomes frequently reflect underpowered evaluation rather than model equivalence.
  arXiv  Detail & Related papers  (2026-01-14T02:34:58Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• Weakly-Supervised Cross-Domain Adaptation for Endoscopic Lesions Segmentation [79.58311369297635]
  We propose a new weakly-supervised lesions transfer framework, which can explore transferable domain-invariant knowledge across different datasets. A Wasserstein quantified transferability framework is developed to highlight widerange transferable contextual dependencies. A novel self-supervised pseudo label generator is designed to equally provide confident pseudo pixel labels for both hard-to-transfer and easy-to-transfer target samples.
  arXiv  Detail & Related papers  (2020-12-08T02:26:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.