SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition

• URL: http://arxiv.org/abs/2608.18303v1
• Date: Tue, 18 Aug 2026 20:27:56 GMT
• Title: SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
• Abstract summary: We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions.<n>On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist.
• Score: 0.30586855806896046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.

Related papers

• JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols [27.45301899307822]
  JUDGESTEALER is a query-efficient model extraction framework for replicating judging capabilities across pointwise scoring, pairwise comparison, and listwise ranking protocols.<n>It exploits the strong cross-protocol agreement to acquire pointwise scores and transform them into pairwise and listwise supervisions without additional victim queries.<n>It consistently outperforms existing extraction baselines, achieving up to 73.3%, 87.0%, and 71.6% accuracy for pointwise, pairwise, and listwise evaluation, respectively.
  arXiv  Detail & Related papers  (2026-08-27T11:28:32Z)
• Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees [16.73004625486562]
  Using LLMs as judges has become standard practice for evaluating model outputs at scale.<n>In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation.<n>We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level.
  arXiv  Detail & Related papers  (2026-08-18T16:42:02Z)
• Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement [13.774572921429217]
  BINEVAL is a framework that decomposes evaluation criteria into atomic binary questions.<n>Given a task prompt, a meta-prompt generates fine-grained evaluation questions.<n>An LLM answers them independently for each output, yielding transparent question-level feedback.
  arXiv  Detail & Related papers  (2026-06-25T16:14:50Z)
• MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following [8.500414706731036]
  We introduce MCJudgeBench, a benchmark for constraint-level judge evaluation in multi-constraint instruction following.<n>Each instance includes an instruction, a candidate response, an explicit constraint list, per-constraint gold labels in yes, partial, no, and controlled response-side perturbations.<n>We evaluate proprietary and open-source LLM judges using both correctness and inconsistency metrics.
  arXiv  Detail & Related papers  (2026-05-05T15:20:42Z)
• Quantifying and Mitigating Self-Preference Bias of LLM Judges [5.017055831476582]
  Self-Preference Bias (SPB) is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation.<n>Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems.<n>We introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences.
  arXiv  Detail & Related papers  (2026-04-24T09:46:22Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• Validating LLM-as-a-Judge Systems under Rating Indeterminacy [65.137380612741]
  We introduce a framework for validating LLM-as-a-judge systems under rating indeterminacy.<n>We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias validation.
  arXiv  Detail & Related papers  (2025-03-07T22:09:47Z)
• Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models [68.92020689188887]
  Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs)<n>Existing automated evaluation metrics cannot fairly evaluate the outputs generated by RAG models during training and evaluation.<n>This paper introduces the Judge-Consistency (ConsJudge) method, which aims to enhance LLMs to generate more accurate evaluations for RAG models.
  arXiv  Detail & Related papers  (2025-02-26T04:50:43Z)
• A Framework for Evaluating LLMs Under Task Indeterminacy [49.298107503257036]
  Large language model (LLM) evaluations often assume there is a single correct response -- a gold label -- for each item in the evaluation corpus. We develop a framework for evaluating LLMs under task indeterminacy.
  arXiv  Detail & Related papers  (2024-11-21T00:15:44Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.