AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

• URL: http://arxiv.org/abs/2603.07019v1
• Date: Sat, 07 Mar 2026 03:36:17 GMT
• Title: AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
• Abstract summary: AutoChecklist is an open-source library that unifies checklist-based evaluation into composable pipelines.<n>A modular Generator $rightarrow$ Refiner $rightarrow$ Scorer pipeline connects any generator with a unified scorer.<n>Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration.
• Score: 16.729112949512807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator $\rightarrow$ Refiner $\rightarrow$ Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.

Related papers

• Evolving Executable Pipeline Programs for AutoML with Language Models [6.598013423021166]
  We present LACE, an AutoML framework that searches over complete executable pipeline programs.<n>An evolutionary loop maintains a population of scikit-learn-compatible Python classes, and a large language model acts as the variation operator.<n>Because every candidate is ordinary Python, the returned pipeline and the search that produced it can be inspected and edited directly.
  arXiv  Detail & Related papers  (2026-08-17T11:15:25Z)
• From Attack Simulation to SIEM Rule: Deterministic Detection-as-Code Synthesis with Probe-Level Traceability [51.56484100374058]
  Security teams simulate attacks against their own systems to check whether their monitoring would catch a real intruder.<n>A human bridges that gap by hand, reading each finding and writing the corresponding Sigma rule.<n>We show this translation can be partially automated when probes are drawn from a locked corpus.
  arXiv  Detail & Related papers  (2026-06-03T14:26:25Z)
• ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks [0.0]
  ActuBench is a pipeline for the automated generation and evaluation of actuarial assessment items.<n>One agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops.<n>The items, per-model responses and complete leaderboard are published as a browsable web interface.
  arXiv  Detail & Related papers  (2026-04-22T07:20:03Z)
• Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning [79.88942231770629]
  Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs)<n>We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR)<n>We synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement.
  arXiv  Detail & Related papers  (2026-04-08T00:53:29Z)
• A Logical-Rule Autoencoder for Interpretable Recommendations [5.768273842489658]
  We propose a Logical-rule Interpretable Autoencoder (LIA) for collaborative filtering that is interpretable by design.<n>LIA introduces a learnable logical rule layer in which each rule neuron is equipped with a gate parameter.<n>LIA allows users to directly trace the decision process behind each recommendation.
  arXiv  Detail & Related papers  (2026-04-05T21:21:37Z)
• CLARC: C/C++ Benchmark for Robust Code Search [2.225731679677886]
  We present CLARC, a C/C++ benchmark built from real-world GitHub repositories.<n>Clarc contains 1,245 query-code pairs for evaluation and 5,472 pairs for training.
  arXiv  Detail & Related papers  (2026-03-04T18:57:37Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• RankLLM: A Python Package for Reranking with LLMs [36.83343408896376]
  This paper introduces RankLLM, an open-source Python package for reranking large language models (LLMs)<n>To improve usability, RankLLM features optional integration with Pyserini for retrieval and provides integrated evaluation for multi-stage pipelines.<n>We reproduce results from RankGPT, LRL, RankVicuna, RankZephyr, and other recent models.
  arXiv  Detail & Related papers  (2025-05-25T19:29:27Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Write Your Own CodeChecker: An Automated Test-Driven Checker Development Approach with LLMs [9.551021559603349]
  AutoChecker is an innovative approach that can write code checkers automatically based on only a rule description and a test suite.<n>Test results show that AutoChecker significantly outperforms others across all effectiveness metrics, with an average test pass rate of 82.28%.
  arXiv  Detail & Related papers  (2024-11-11T08:50:24Z)
• Self-Calibrated Listwise Reranking with Large Language Models [137.6557607279876]
  Large language models (LLMs) have been employed in reranking tasks through a sequence-to-sequence approach. This reranking paradigm requires a sliding window strategy to iteratively handle larger candidate sets. We propose a novel self-calibrated listwise reranking method, which aims to leverage LLMs to produce global relevance scores for ranking.
  arXiv  Detail & Related papers  (2024-11-07T10:31:31Z)
• AlignBench: Benchmarking Chinese Alignment of Large Language Models [99.24597941555277]
  We introduce AlignBench, a comprehensive benchmark for evaluating Chinese Large Language Models' alignment. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judgecitezheng2023judging approach with Chain-of-Thought to generate explanations and final ratings.
  arXiv  Detail & Related papers  (2023-11-30T17:41:30Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• A multi-language toolkit for the semi-automated checking of research outputs [0.44998333629984877]
  This article presents a free and open source toolkit that supports the semi-automated checking of research outputs (SACRO) for privacy disclosure within secure data environments.<n>SACRO is a framework that applies best-practice principles-based statistical disclosure control (SDC) techniques on-the-fly as researchers conduct their analyses.<n>The toolkit is composed of a lightweight Python package that sits over well-known analysis tools that produce outputs such as tables, plots, and statistical models.
  arXiv  Detail & Related papers  (2022-12-06T12:45:15Z)
• SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics [74.28810048824519]
  SacreROUGE is an open-source library for using and developing summarization evaluation metrics. The library provides Python wrappers around the official implementations of existing evaluation metrics. It provides functionality to evaluate how well any metric implemented in the library correlates to human-annotated judgments.
  arXiv  Detail & Related papers  (2020-07-10T13:26:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.