Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

• URL: http://arxiv.org/abs/2606.24965v1
• Date: Tue, 23 Jun 2026 09:36:46 GMT
• Title: Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners
• Abstract summary: Reasoning about relational structures remains a significant challenge for neural models.<n>We study how this issue can be addressed by using large language models to automate benchmark generation.<n>We show that the same machinery can be applied to novel worlds proposed by LLMs, opening the door to autonomous research on neural relational reasoning.
• Score: 18.516862091292328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning about relational structures remains a significant challenge for neural models, particularly when they must systematically apply learned knowledge to problem instances that are harder than those seen in training. Progress is hampered by the difficulty of evaluating such generalization, since a priori, it is rarely clear what makes an instance hard. We study how this issue can be addressed by using large language models (LLMs) to automate benchmark generation, learning to produce increasingly challenging instances in an end-to-end manner. Concretely, given a world parametrized by Datalog rules, and an Edge Transformer as the reasoning evaluator, we use LLM-driven evolutionary search (based on FunSearch) and autonomous agentic search to discover sampling functions that yield hard problem instances. We also show that the Edge Transformer can be improved using this data such that it generalizes well to further data perturbations. Finally, we show that the same machinery can be applied to novel worlds proposed by LLMs, opening the door to autonomous research on neural relational reasoning.

Related papers

• Towards Compositional Generalization in LLMs for Smart Contract Security: A Case Study on Reentrancy Vulnerabilities [35.39583123277091]
  This paper proposes a post-training algorithm based on atomic task decomposition and fusion.<n>We decompose the reentrancy vulnerability detection task into four linearly independent atomic tasks.<n>By training on synthetic datasets, we generate three compiler-verified datasets.<n>We then employ the Slither tool to extract structural information from the control flow graph and data flow graph.
  arXiv  Detail & Related papers  (2026-01-11T13:52:07Z)
• FOL-Pretrain: A complexity annotated corpus of first-order logic [16.061040115094592]
  Transformer-based large language models (LLMs) have demonstrated remarkable reasoning capabilities.<n>Despite recent efforts to reverse-engineer LLM behavior, our understanding of how these models internalize and execute complex algorithms remains limited.<n>We introduce a large-scale, fully open, complexity-annotated dataset of first-order logic reasoning traces.
  arXiv  Detail & Related papers  (2025-05-20T21:38:28Z)
• When Do Neural Networks Learn World Models? [8.414327163725622]
  We present the first theoretical results for whether neural networks can learn similar world models.<n>We show that models with a low-degree bias provably recover latent data-generating variables under mild assumptions.
  arXiv  Detail & Related papers  (2025-02-13T13:11:54Z)
• Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models [31.769428095250912]
  Auto-RAG is an autonomous iterative retrieval model centered on the reasoning capabilities of Large Language Models (LLMs)<n>We develop a method for autonomously synthesizing reasoning-based decision-making instructions in iterative retrieval.<n>Auto-RAG expresses the iterative retrieval process in natural language, enhancing interpretability.
  arXiv  Detail & Related papers  (2024-11-29T03:01:05Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• Auto-FP: An Experimental Study of Automated Feature Preprocessing for Tabular Data [10.740391800262685]
  Feature preprocessing is a crucial step to ensure good model quality. Due to the large search space, a brute-force solution is prohibitively expensive. We extend a variety of HPO and NAS algorithms to solve the Auto-FP problem.
  arXiv  Detail & Related papers  (2023-10-04T02:46:44Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• Automatic Data Augmentation via Invariance-Constrained Learning [94.27081585149836]
  Underlying data structures are often exploited to improve the solution of learning tasks. Data augmentation induces these symmetries during training by applying multiple transformations to the input data. This work tackles these issues by automatically adapting the data augmentation while solving the learning task.
  arXiv  Detail & Related papers  (2022-09-29T18:11:01Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Amortized Inference for Causal Structure Learning [72.84105256353801]
  Learning causal structure poses a search problem that typically involves evaluating structures using a score or independence test. We train a variational inference model to predict the causal structure from observational/interventional data. Our models exhibit robust generalization capabilities under substantial distribution shift.
  arXiv  Detail & Related papers  (2022-05-25T17:37:08Z)
• Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness [68.97830259849086]
  Most datasets only capture a simpler subproblem and likely suffer from spurious features. We study adversarial robustness - a local generalization property - to reveal hard, model-specific instances and spurious features. Unlike in other applications, where perturbation models are designed around subjective notions of imperceptibility, our perturbation models are efficient and sound. Surprisingly, with such perturbations, a sufficiently expressive neural solver does not suffer from the limitations of the accuracy-robustness trade-off common in supervised learning.
  arXiv  Detail & Related papers  (2021-10-21T07:28:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.