NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation

• URL: http://arxiv.org/abs/2608.07530v1
• Date: Fri, 24 Jul 2026 10:25:35 GMT
• Title: NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation
• Abstract summary: We present NL2SHACL-Bench, a benchmark suite for natural language to SHACL translation.<n>Using NL2SHACL-Bench, we evaluate four state-of-the-art large language models (LLMs) for this task.
• Score: 2.8949978680533075
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: SHACL is a core technology for validating the conformance of RDF knowledge graphs (KGs). Yet, authoring SHACL shapes requires technical expertise that most domain experts lack. Translating natural language requirements into SHACL (NL2SHACL) would lower this barrier. However, there is no dedicated benchmark for NL2SHACL, and evaluating generated shapes requires methods beyond string comparison, as semantically equivalent shapes can differ in serialisation and structure. To tackle these challenges, we present NL2SHACL-Bench, a benchmark suite for natural language to SHACL translation. Using NL2SHACL-Bench, we evaluate four state-of-the-art large language models (LLMs) for this task. Our results show that current LLMs are highly capable of generating syntactically valid SHACL, but still struggle to produce semantically equivalent constraints for complex logical and structural patterns. This indicates that NL2SHACL-Bench provides a meaningful basis for measuring advances in the NL2SHACL state of the art.

Related papers

• SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging [49.898541267264115]
  Maintenance is increasingly challenging due to outdated technologies, sparse documentation, and developer retirement.<n>Traditional rule-based transcompilers yield outputs that are difficult to read and maintain.<n>We propose SEDCoT, a novel-to-C translation framework.
  arXiv  Detail & Related papers  (2026-07-05T03:07:10Z)
• NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming [62.89531126732269]
  We introduce NL2Scratch, an executable benchmark for natural-language-to-Scratch generation.<n>We construct a semantically validated pool of 23,594 examples, and a slot-balanced 800 diagnostic benchmark.<n>Experiments across instruction-tuned and fine-tuned LLMs reveal a notable gap between lexical similarity and semantic alignment.
  arXiv  Detail & Related papers  (2026-06-20T14:22:59Z)
• SHACL Validation in the Presence of Ontologies: Semantics and Rewriting Techniques [6.882042556551609]
  SHACL is a constraint language that treats the data as complete and must be validated under the closed-world assumption.<n>We advocate a semantics for SHACL validation in the presence of major core universal models.<n>We show that even very simple features make the problem EXPTIME-complete, and PTIME-complete in data complexity.
  arXiv  Detail & Related papers  (2025-07-16T14:38:27Z)
• xpSHACL: Explainable SHACL Validation using Retrieval-Augmented Generation and Large Language Models [0.0]
  Shapes Constraint Language (SHACL) is a powerful language for validating RDF data.<n>This paper presents XPSHACL, an explainable SHACL validation system.<n>It combines rule-based justification trees with retrieval-augmented generation (RAG) and large language models (LLMs) to produce detailed, multilanguage explanations for constraint violations.
  arXiv  Detail & Related papers  (2025-07-11T09:18:41Z)
• BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models [50.17907898478795]
  We introduce BinMetric, a benchmark designed to evaluate the performance of large language models on binary analysis tasks.<n>BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks.<n>Our empirical study on this benchmark investigates the binary analysis capabilities of various state-of-the-art LLMs, revealing their strengths and limitations in this field.
  arXiv  Detail & Related papers  (2025-05-12T08:54:07Z)
• SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition [77.28814034644287]
  We propose SVTRv2, a CTC model endowed with the ability to handle text irregularities and model linguistic context.<n>We extensively evaluate SVTRv2 in both standard and recent challenging benchmarks.<n> SVTRv2 surpasses most EDTRs across the scenarios in terms of accuracy and inference speed.
  arXiv  Detail & Related papers  (2024-11-24T14:21:35Z)
• Bayesian scaling laws for in-context learning [85.34114399339741]
  In-context learning (ICL) is a powerful technique for getting language models to perform complex tasks with no training updates.<n>We show that ICL approximates a Bayesian learner, which gives rise to a novel Bayesian scaling law for ICL.<n>Our scaling law matches existing scaling laws in accuracy while also offering interpretable terms for task priors, learning efficiency, and per-example probabilities.
  arXiv  Detail & Related papers  (2024-10-21T21:45:22Z)
• Augmented Language Models: a Survey [55.965967655575454]
  This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. We refer to them as Augmented Language Models (ALMs) The missing token objective allows ALMs to learn to reason, use tools, and even act, while still performing standard natural language tasks.
  arXiv  Detail & Related papers  (2023-02-15T18:25:52Z)
• Improving Bilingual Lexicon Induction with Cross-Encoder Reranking [31.142790337451366]
  We propose a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking) The key idea is to 'extract' cross-lingual lexical knowledge from mPLMs, and then combine it with the original CLWEs. BLICEr establishes new results on two standard BLI benchmarks spanning a wide spectrum of diverse languages.
  arXiv  Detail & Related papers  (2022-10-30T21:26:07Z)
• Satisfiability and Containment of Recursive SHACL [4.8986598953553555]
  The Shapes Constraint Language (SHACL) is the recent W3C recommendation language for validating RDF data, by verifying certain shapes on graphs. Previous work has largely focused on the validation problem and the standard decision problems of satisfiability and containment. We provide a comprehensive study of the different features of SHACL, by providing a translation to a new first-order language, called SCL, that precisely captures the semantics of SHACL. We also present MSCL, a second-order extension of SCL, which allows us to define, in a single formal logic framework, the main
  arXiv  Detail & Related papers  (2021-08-30T08:51:03Z)
• SHACL Satisfiability and Containment (Extended Paper) [6.308539010172308]
  The Shapes Constraint Language (SHACL) is a recent W3C recommendation language for validating RDF data. In this paper, we undertake a thorough study of different features of non-recursive SHACL by providing a translation to a new first-order language, called SCL. We study the interaction of SHACL features in this logic and provide the detailed map of decidability and complexity results of the aforementioned decision problems for different SHACL sublanguages.
  arXiv  Detail & Related papers  (2020-08-31T14:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.