PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

• URL: http://arxiv.org/abs/2606.08481v1
• Date: Sun, 07 Jun 2026 06:53:09 GMT
• Title: PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems
• Abstract summary: PIPE-Cypher is a benchmark-generation pipeline that turns a live property graph and optional seed queries into balanced NL-to-Cypher benchmarks.<n>We show how PIPE-Cypher exports 3,000 accepted FinBench/SNB examples, completes three audited ablation suites, calibrates judge behavior with human labels, and evaluates 11 local downstream models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enterprise property graphs vary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. A deployment-relevant Text2Cypher benchmark therefore reflects the questions users and agents actually ask of that graph. Creating such a benchmark is difficult because schemas and values are unique, and graph structure changes over time. Each NL-query pair must also be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. We present PIPE-Cypher, a local benchmark-generation pipeline that turns a live property graph and optional seed queries from customer questions, analyst logs, or agent tool calls into balanced NL-to-Cypher benchmarks. PIPE-Cypher combines schema profiling, reverse-query grounding, constrained generation, deterministic Cypher governance, execution validation, redaction, diversity controls, and a calibrated local LLM judge. Using local Qwen3.5-9B generation and judging, PIPE-Cypher exports 3,000 accepted FinBench/SNB examples, completes three audited ablation suites, calibrates judge behavior with human labels, and evaluates 11 local downstream models. The resulting benchmark is deliberately discriminative: zero-shot transfer is weak, while a few-shot control shows that schema-specific example banks can help compatible model families. Together, PIPE-Cypher makes Text2Cypher benchmarking a repeatable process that evolves with the graph, its users, and its target workloads.

Related papers

• LoGIC: Budgeted Context Construction for Node-Level Graph In-Context Learning with Tabular Foundation Models [2.895347325484886]
  Current protocols employ the complete training table as context, causing attention to scale quadratically with the labeled pool and introducing preprocessing and memory bottlenecks.<n>We investigate context construction for node-level graph ICL: which labeled nodes and auxiliary unlabeled nodes should constitute the prompt for specified queries.<n>We present LoGIC, which retrieves labeled nodes via structural, feature-based, and coverage channels, shares each context across the queries in a graph-local cluster, incorporates an unlabeled halo for adapter backbones, and chooses the channel and context budget without test labels.
  arXiv  Detail & Related papers  (2026-09-05T07:43:30Z)
• BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click [0.0]
  We present BekchiAI, a benchmark for measuring agentic skill and a platform for observing and controlling live agents.<n>The BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and telemetry as well as remote run termination.
  arXiv  Detail & Related papers  (2026-08-27T09:30:08Z)
• Who Belongs in the Eval Set? A Capability-Taxonomy-Driven Pipeline for Curating Regression Eval Sets in Agent-Extensibility Platforms [1.7499351967216341]
  We describe a capability-taxonomy-driven curation pipeline applied to declarative agents with custom actions in Microsoft 365 Copilot.<n>It takes an agent specification and a customer's eval set as input, projects each query into a platform-owned capability taxonomy.<n>The mechanism is taxonomy-agnostic and applies to any regression eval-set problem with a typed capability taxonomy.
  arXiv  Detail & Related papers  (2026-08-02T05:19:25Z)
• BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning [0.8681424810073922]
  BADGER is a unified framework integrating text-to-assessment with agentic behavior evaluation.<n> BADGER runs entirely within client's governed data environment.<n>It supports rapid prototyping of client-specific judges and metrics.
  arXiv  Detail & Related papers  (2026-06-01T11:42:37Z)
• TraceGraph: Shared Decision Landscapes for Diagnosing and Improving Agent Trajectories [61.51984029109709]
  We introduce TraceGraph, a graph-based framework that turns released multi-model agent trajectories into shared decision landscapes.<n>For each task, TraceGraph builds a graph over observable action-observation states from pooled rollouts before model identity is introduced.<n>It then overlays outcome-informed productive cores and trap regions, and summarizes each rollout with three events: Access, Trap exposure, and Repair.
  arXiv  Detail & Related papers  (2026-05-29T13:40:31Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement [10.675982676332941]
  Existing approaches treat schemas as fixed and address errors downstream.<n>We frame schema refinement as a constrained optimization problem.<n>We analyze the computational hardness of this problem, which motivates a column-wise greedy decomposition.
  arXiv  Detail & Related papers  (2026-05-01T13:01:59Z)
• AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesis [30.512393568258105]
  Large Language Model agents demonstrate potential in solving real-world problems via tools, yet generalist intelligence is bottlenecked by scarce high-quality, long-horizon data.<n>We propose AgentSkiller, a fully automated framework synthesizing multi-turn interaction data across realistic, semantically linked domains.
  arXiv  Detail & Related papers  (2026-02-10T03:21:42Z)
• PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech [0.0]
  ProfASR-Bench is a professional-talk evaluation suite for high-stakes applications across finance, medicine, legal, and technology.<n>Each example pairs a natural-language prompt with an entity-rich target utterance, enabling controlled measurement of context-conditioned recognition.
  arXiv  Detail & Related papers  (2025-12-29T18:43:23Z)
• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering [64.62317305868264]
  We present textbfKBQA-R1, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning.<n>Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions.<n>Experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-10T17:45:42Z)
• Multi-Agent GraphRAG: A Text-to-Cypher Framework for Labeled Property Graphs [7.943264761730892]
  Multi-Agent GraphRAG serves as a natural language interface to LPG-based graph data.<n>Iterative content-aware correction and normalization, reinforced by an aggregated feedback loop, ensures both semantic and syntactic refinement of generated queries.<n>This highlights how such an approach can bridge AI with real-world applications at scale, enabling industrial digital automation use cases.
  arXiv  Detail & Related papers  (2025-11-11T14:04:00Z)
• Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents [106.04963073116468]
  Flowcharts are a critical tool for visualizing decision-making processes.<n> vision-language models frequently hallucinate nonexistent connections and decision paths when analyzing these diagrams.<n>We introduce Fine-grained Flowchart, which traces specific components grounding a flowchart referring LLM response.<n>We propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post hoc attribution through graph-based reasoning.
  arXiv  Detail & Related papers  (2025-06-02T06:02:41Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking [55.62705574507595]
  OpenPI is the only dataset annotated for open-vocabulary state tracking. We categorize 3 types of problems on the procedure level, step level and state change level respectively. For the evaluation metric, we propose a cluster-based metric to fix the original metric's preference for repetition.
  arXiv  Detail & Related papers  (2023-06-01T16:48:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.