Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications

• URL: http://arxiv.org/abs/2607.14673v1
• Date: Thu, 16 Jul 2026 07:38:39 GMT
• Title: Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications
• Abstract summary: We present Kaleidoscope, an integrated workflow for contextual functional evaluation.<n>It links persona-based test generation, contextualized rubrics, and human review for reliability-gated automated scoring.<n>We report early evidence from a three-week pilot across four organizational use cases and custom-rubric judge experiments.
• Score: 9.941992941056993
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluations (Evals) are a deployment bottleneck for real-world AI applications: public benchmarks rarely match a team's users, context, or policies, and human review is often tedious to scale. Motivated by our work with AI applications in the public sector, this project addresses recurring evaluation challenges encountered when applications must satisfy local policy and governance requirements. We present Kaleidoscope, an integrated workflow for contextual functional evaluation that links persona-based test generation, contextualized rubrics, and human review for reliability-gated automated scoring. Generated test cases are scored against application-specific rubrics; human annotations provide reviewable labels; and LLM judges automate scoring only when their agreement with those labels meets a configured threshold. Kaleidoscope is therefore a practical, inspectable, iterative workflow for product teams. We report early evidence from a three-week pilot across four organizational use cases and custom-rubric judge experiments on 108 annotated Q\&A pairs spanning four domains and 14 evaluation dimensions. The results highlight useful features for end-to-end reliable, automated scoring.

Related papers

• DualEval: Joint Model-Item Calibration for Unified LLM Evaluation [67.99119926012686]
  DualEval is a latent model-item calibration framework that represents models and evaluation items in a shared space.<n>We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries.
  arXiv  Detail & Related papers  (2026-06-24T22:40:46Z)
• AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility [104.46861849039357]
  Agent systems are advancing quickly across domains, but their evaluation remains fragmented.<n>The root problem is the lack of an open, agent-agnostic assessment interface.<n>We advocate Agentified Agent Assessment (AAA), where evaluation is performed by judge agents and all participants interact through standardized protocols.
  arXiv  Detail & Related papers  (2026-06-11T17:23:54Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs [48.36707944399574]
  We study how vibe-testing works in practice and then formalize it to support systematic analysis.<n>In experiments on coding benchmarks, we find that combining personalized prompts and user-aware evaluation can change which model is preferred.
  arXiv  Detail & Related papers  (2026-04-15T17:57:08Z)
• One-Eval: An Agentic System for Automated and Traceable LLM Evaluation [10.701916838477187]
  One-Eval is an agentic evaluation system that converts natural-language requests into executable evaluations.<n>One-Eval supports more efficient and reproducible evaluation in industrial settings.
  arXiv  Detail & Related papers  (2026-03-10T15:45:51Z)
• Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark [9.922581736690159]
  We present a large-scale empirical study of AI grading on real, handwritten calculus work from UC Irvine.<n>Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of free-response quiz submissions.<n>In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review.
  arXiv  Detail & Related papers  (2026-03-01T03:32:51Z)
• Principled Design of Interpretable Automated Scoring for Large-Scale Educational Assessments [2.2219355720968967]
  AnalyticScore operates by extracting explicitly identifiable elements of the responses and featurizing each response into human-interpretable values.<n>AnalyticScore is within only 0.06 QWK of the uninterpretable SOTA on average across 10 items from the ASAP-SAS dataset.
  arXiv  Detail & Related papers  (2025-11-21T09:19:05Z)
• What Users Value and Critique: Large-Scale Analysis of User Feedback on AI-Powered Mobile Apps [2.352412885878654]
  We present the first comprehensive, large-scale study of user feedback on AI-powered mobile apps.<n>We leverage a curated dataset of 292 AI-driven apps across 14 categories with 894K AI-specific reviews from Google Play.<n>Our pipeline surfaces both satisfaction with one feature and frustration with another within the same review.
  arXiv  Detail & Related papers  (2025-06-12T14:56:52Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain.<n>Our benchmark is characterized by its multi-dimensional evaluation framework.<n>Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI [0.0]
  We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Our findings underscore the importance of a human-centered workflow and careful evaluation standards.
  arXiv  Detail & Related papers  (2024-09-14T15:27:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.