CAFE: A Compound-AI Factorial Evaluation Framework

• URL: http://arxiv.org/abs/2607.10380v1
• Date: Sat, 11 Jul 2026 16:17:46 GMT
• Title: CAFE: A Compound-AI Factorial Evaluation Framework
• Abstract summary: We introduceCompound-AI Factorial Evaluation (CEFA), an open-source platform that brings design of experiments to the evaluation of compound AI systems (CAIS)<n>CEFA registers each swappable component of a pipeline as a factor to build a factorial design over the chosen factors.<n>It attributes answer-quality variance to the components and their interactions with mixed-effects models and reports effect sizes, significance, the best configuration, cost and latency trade-offs, and judge-human reliability.
• Score: 40.35767028179069
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce CAFE (Compound-AI Factorial Evaluation), an open-source platform that brings design of experiments to the evaluation of compound AI systems (CAIS). Such systems expose many interchangeable choices - e.g. which retriever, model, or prompt - and practitioners rarely know which of them most affects answer quality. With CAFE, a practitioner registers each swappable component of a pipeline as a factor to build a factorial design over the chosen factors, run the resulting configurations, and score the answers on a shared rubric using a configurable LLM judge together with human raters. From these ratings it attributes answer-quality variance to the components and their interactions with mixed-effects models and reports effect sizes, significance, the best configuration, cost and latency trade-offs, and judge-human reliability. Whereas existing tools mostly either search for a good configuration or score outputs in isolation, CAFE also explains which component drives quality and whether an observed difference is significant. We validate CAFE on a retrieval-augmented question-answering (QA) pipeline over the HotpotQA benchmark dataset, where it recovers planted factor effects and stays calibrated under a permutation null. CAFE is released as a Python package and as a Web application.

Related papers

• Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control [59.15149762610843]
  Reinforcement learning systems are significantly more complex than other machine learning paradigms.<n>Despite advances in individual algorithmic components, their functional interdependencies remain underexplored.<n>We propose ROSER, an RL framework that coordinates three critical dimensions: Model-based Representation, Optimization Stability, and Experience Replay.
  arXiv  Detail & Related papers  (2026-08-07T10:38:31Z)
• DualEval: Joint Model-Item Calibration for Unified LLM Evaluation [67.99119926012686]
  DualEval is a latent model-item calibration framework that represents models and evaluation items in a shared space.<n>We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries.
  arXiv  Detail & Related papers  (2026-06-24T22:40:46Z)
• Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies [0.5352699766206808]
  This research aims to better understand how multi-hop reasoning can be most effectively evaluated in RAG systems.<n>Experiments with LLMs from OpenAI, Meta, and Google demonstrate that CARE consistently outperforms existing methods for evaluating multi-hop reasoning in RAG systems.
  arXiv  Detail & Related papers  (2026-04-20T13:20:57Z)
• PCoKG: Personality-aware Commonsense Reasoning with Debate [32.49722822521962]
  Personality-aware Commonsense Knowledge Graph (PCoKG) is a structured dataset comprising 521,316 quadruples.<n>For knowledge graph construction, we leverage the role-playing capabilities of large language models.<n>We apply PCoKG to persona-based dialogue generation, where it demonstrates improved consistency between generated responses and reference outputs.
  arXiv  Detail & Related papers  (2026-01-09T15:05:01Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning [88.55095746156428]
  Retrieval-augmented generation (RAG) is widely utilized to incorporate external knowledge into large language models.<n>A standard RAG pipeline consists of several components, such as query rewriting, document retrieval, document filtering, and answer generation.<n>We propose treating the complex RAG pipeline with multiple components as a multi-agent cooperative task, in which each component can be regarded as an RL agent.
  arXiv  Detail & Related papers  (2025-01-25T14:24:50Z)
• Compositional Models for Estimating Causal Effects [0.9208007322096533]
  We study a compositional approach for estimating individual-level potential outcomes and causal effects in structured systems.<n>The compositional approach decomposes unit-level causal queries into more fine-grained queries.<n>We show that it provides benefits for causal effect estimation from observational data.
  arXiv  Detail & Related papers  (2024-06-25T16:56:17Z)
• Improving Text Matching in E-Commerce Search with A Rationalizable, Intervenable and Fast Entity-Based Relevance Model [78.80174696043021]
  We propose a novel model called the Entity-Based Relevance Model (EBRM) The decomposition allows us to use a Cross-encoder QE relevance module for high accuracy. We also show that pretraining the QE module with auto-generated QE data from user logs can further improve the overall performance.
  arXiv  Detail & Related papers  (2023-07-01T15:44:53Z)
• Fairness and underspecification in acoustic scene classification: The case for disaggregated evaluations [6.186191586944725]
  Underspecification and fairness in machine learning (ML) applications have recently become two prominent issues in the ML community. We argue for the need of a more holistic evaluation process for Acoustic scene classification (ASC) models through disaggregated evaluations. We demonstrate the effectiveness of the proposed evaluation process in uncovering underspecification and fairness problems when trained on two widely-used ASC datasets.
  arXiv  Detail & Related papers  (2021-10-04T15:23:01Z)
• Contextualized Embeddings based Convolutional Neural Networks for Duplicate Question Identification [0.25782420501870296]
  Question Paraphrase Identification (QPI) is a critical task for large-scale Question-Answering forums. We propose a novel architecture combining a Bidirectional Transformer with Convolutional Neural Networks for the QPI task. Experimental results demonstrate that our model achieves state-of-the-art performance on the Quora Question Pairs dataset.
  arXiv  Detail & Related papers  (2021-09-03T14:30:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.