Semantic Operators: A Declarative Model for Rich, AI-based Data Processing

• URL: http://arxiv.org/abs/2407.11418v3
• Date: Sat, 01 Mar 2025 01:47:51 GMT
• Title: Semantic Operators: A Declarative Model for Rich, AI-based Data Processing
• Abstract summary: We introduce semantic operators, the first formalism for declarative and general-purpose AI-based transformations.<n>We propose several novel optimizations to accelerate semantic filtering, joining, group-by and top-k operations by up to $1,000times$.<n>We show that the semantic operator model is expressive, capturing state-of-the-art AI pipelines in a few operator calls.
• Score: 27.294714926492187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The semantic capabilities of large language models (LLMs) have the potential to enable rich analytics and reasoning over vast knowledge corpora. Unfortunately, existing systems either empirically optimize expensive LLM-powered operations with no performance guarantees, or serve a limited set of row-wise LLM operations, providing limited robustness, expressiveness and usability. We introduce semantic operators, the first formalism for declarative and general-purpose AI-based transformations based on natural language specifications (e.g., filtering, sorting, joining or aggregating records using natural language criteria). Each operator opens a rich space for execution plans, similar to relational operators. Our model specifies the expected behavior of each operator with a high-quality gold algorithm, and we develop an optimization framework that reduces cost, while providing accuracy guarantees with respect to a gold algorithm. Using this approach, we propose several novel optimizations to accelerate semantic filtering, joining, group-by and top-k operations by up to $1,000\times$. We implement semantic operators in the LOTUS system and demonstrate LOTUS' effectiveness on real, bulk-semantic processing applications, including fact-checking, biomedical multi-label classification, search, and topic analysis. We show that the semantic operator model is expressive, capturing state-of-the-art AI pipelines in a few operator calls, and making it easy to express new pipelines that match or exceed quality of recent LLM-based analytic systems by up to $170\%$, while offering accuracy guarantees. Overall, LOTUS programs match or exceed the accuracy of state-of-the-art AI pipelines for each task while running up to $3.6\times$ faster than the highest-quality baselines. LOTUS is publicly available at https://github.com/lotus-data/lotus.

Related papers

• DGA$_2$D: Directed Graph-Guided Automated Algorithm Design with Large Language Models [1.8062322656999443]
  This paper proposes a Directed Graph-Guided Automated Design framework, DGA$D.<n>It structures the open-ended program space as a directed graph, where each node represents a functional operator that can be instantiated.<n>It reduces the average normalized gap by up to 10.96 percentage points compared to state-of-the-art LLM baselines.
  arXiv  Detail & Related papers  (2026-08-01T14:59:38Z)
• FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization [61.43300970020897]
  Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation.<n>Existing benchmarks are limited to small or simplified examples far below real-world scale and complexity.<n>We introduce FrontierOR, among the first benchmarks to evaluate LLM-based efficient algorithm design for realistic large-scale optimization problems.
  arXiv  Detail & Related papers  (2026-05-24T20:10:42Z)
• Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs [1.2957535240267326]
  Large Language Models (LLMs) have demonstrated strong capabilities in natural language understanding and reasoning.<n>We systematically evaluate multiple prompting strategies, including Chain-of-Thought (CoT), Least-to-Most decomposition, Program-of-Thought (PoT), and Self-Consistency (SC)<n>Our results show that standard prompting methods achieve only moderate accuracy on sequence-based tasks.
  arXiv  Detail & Related papers  (2026-05-04T23:32:37Z)
• Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization [35.60979104539273]
  Nirvana is a multi-modal data analytics framework that incorporates programmable semantic operators.<n>Nirvana is able to reduce end-to-end runtime by 10%--85% and reduces system processing costs by 76% on average.
  arXiv  Detail & Related papers  (2025-11-25T01:41:49Z)
• Toward a Trustworthy Optimization Modeling Agent via Verifiable Synthetic Data Generation [11.988926173584154]
  We present a framework for training trustworthy large language model (LLM) agents via a synthetic data generation pipeline.<n> OptiTrust is a modular LLM agent that performs multi-language translation from natural language to solver-ready code.<n>Our agent achieves state-of-the-art performance on standard benchmarks.
  arXiv  Detail & Related papers  (2025-08-05T05:54:20Z)
• Evaluating Large Language Models on Non-Code Software Engineering Tasks [4.381476817430934]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code understanding and generation.<n>We present the first comprehensive benchmark, which we name Software Engineering Language Understanding' (SELU)<n>SELU covers classification, regression, Named Entity Recognition (NER) and Masked Language Modeling (MLM) targets, with data drawn from diverse sources.
  arXiv  Detail & Related papers  (2025-06-12T15:52:32Z)
• Abacus: A Cost-Based Optimizer for Semantic Operator Systems [12.831601498212397]
  We present Abacus, a cost-based system that searches for the best implementation of a semantic operator system.<n>Abacus estimates operator performance by leveraging a minimal set of validation examples and, if available, prior beliefs about operator performance.<n>We demonstrate that systems optimized by Abacus achieve 19.2%-39.2% better quality and up to 23.6x lower cost and 4.2x lower latency than the next best system.
  arXiv  Detail & Related papers  (2025-05-20T17:49:46Z)
• Decision Tree Induction Through LLMs via Semantically-Aware Evolution [53.0367886783772]
  We propose an evolutionary optimization method for decision tree induction based on genetic programming (GP) Our key innovation is the integration of semantic priors and domain-specific knowledge about the search space into the algorithm. This is operationalized through novel genetic operators that work with structured natural language prompts.
  arXiv  Detail & Related papers  (2025-03-18T12:52:03Z)
• Language Models can Self-Improve at State-Value Estimation for Better Search [23.61729554517216]
  We introduce Self-Taught Lookahead (STL), a reward-free framework that improves language model-based value functions by reasoning explicitly about state transitions.<n>We find that STL enables small open-source models to guide efficient search, reducing inference costs by integrating explicit reasoning with value learning.
  arXiv  Detail & Related papers  (2025-03-04T18:58:11Z)
• $\texttt{SEM-CTRL}$: Semantically Controlled Decoding [53.86639808659575]
  $texttSEM-CTRL$ is a unified approach that enforces rich context-sensitive constraints and task- and instance-specific semantics directly on an LLM decoder. texttSEM-CTRL$ allows small pre-trained LLMs to efficiently outperform larger variants and state-of-the-art reasoning models.
  arXiv  Detail & Related papers  (2025-03-03T18:33:46Z)
• SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors [5.247363735860479]
  Large language models (LLMs) have demonstrated remarkable capabilities in code-related tasks. Given LLMs' ability to understand and process diverse programs, they present a promising direction for building general-purpose surrogate models. We introduce SURGE, a benchmark with $1160$ problems covering $8$ key aspects. Through empirical analysis of $21$ open-source and proprietary LLMs, we examine scaling laws, data efficiency, and predictive accuracy.
  arXiv  Detail & Related papers  (2025-02-16T15:38:19Z)
• Autoformulation of Mathematical Optimization Models Using LLMs [50.030647274271516]
  This paper approaches the problem of $textitautoformulation$: the automated creation of solver-ready optimization models from natural language problem descriptions.<n>We identify three core challenges of autoformulation: $textit(1)$ the vast, problem-dependent hypothesis space, and $textit(2)$ efficient and diverse exploration of this space under uncertainty.<n>We present a novel method leveraging $textitLarge Language Models$ with $textitMonte-Carlo Tree Search$, exploiting the hierarchical nature of optimization modeling to generate and systematically explore possible formulations
  arXiv  Detail & Related papers  (2024-11-03T20:41:38Z)
• $f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization [54.94545757220999]
  $f$-PO is a novel framework that generalizes and extends existing approaches. We conduct experiments on state-of-the-art language models using benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-29T02:11:45Z)
• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• ToolACE: Winning the Points of LLM Function Calling [139.07157814653638]
  ToolACE is an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data. We demonstrate that models trained on our synthesized data, even with only 8B parameters, achieve state-of-the-art performance on the Berkeley Function-Calling Leaderboard.
  arXiv  Detail & Related papers  (2024-09-02T03:19:56Z)
• Efficient Budget Allocation for Large-Scale LLM-Enabled Virtual Screening [0.9558392439655016]
  We consider an LLM-as-human-evaluator approach for conducting screening virtually, thereby reducing the cost burden. We propose using a top-$m$ greedy evaluation mechanism, and design the explore-first top-$m$ greedy (EFG-$m$) algorithm. Surprisingly, we uncover a bonus ranking effect, where the algorithm naturally induces an indifference-based ranking within the selected subset.
  arXiv  Detail & Related papers  (2024-08-18T16:44:41Z)
• UQE: A Query Engine for Unstructured Databases [71.49289088592842]
  We investigate the potential of Large Language Models to enable unstructured data analytics. We propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.
  arXiv  Detail & Related papers  (2024-06-23T06:58:55Z)
• MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic [6.46176287368784]
  We propose textbfModel textbfExclusive textbfTask textbfArithmetic for merging textbfGPT-scale models. Our proposed MetaGPT is data-agnostic and bypasses the heavy search process, making it cost-effective and easy to implement for LLMs.
  arXiv  Detail & Related papers  (2024-06-17T10:12:45Z)
• IQLS: Framework for leveraging Metadata to enable Large Language Model based queries to complex, versatile Data [0.20482269513546458]
  The Intelligent Query and Learning System (IQLS) simplifies the process by allowing natural language use to simplify data retrieval. It maps structured data into a framework based on the available metadata and available data models. The IQLS enables the agent to fulfill tasks given by the user query through interfaces.
  arXiv  Detail & Related papers  (2024-05-04T13:44:05Z)
• Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers [66.823588073584]
  Large language models (LLMs) have shown remarkable instruction-following capabilities and achieved impressive performances in various applications. Recent work has used the query-efficient Bayesian optimization (BO) algorithm to automatically optimize the instructions given to black-box LLMs. We propose a neural bandit algorithm which replaces the GP in BO by an NN surrogate to optimize instructions for black-box LLMs.
  arXiv  Detail & Related papers  (2023-10-02T02:01:16Z)
• FIND: A Function Description Benchmark for Evaluating Interpretability Methods [86.80718559904854]
  This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. We evaluate methods that use pretrained language models to produce descriptions of function behavior in natural language and code.
  arXiv  Detail & Related papers  (2023-09-07T17:47:26Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs [66.30706841821123]
  Large language models (LLMs) power many state-of-the-art systems in natural language processing. LLMs are extremely computationally expensive, even at inference time. We propose a new metric for comparing inference efficiency across models.
  arXiv  Detail & Related papers  (2023-05-03T21:51:42Z)
• Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes [54.13559879916708]
  EVAPORATE is a prototype system powered by large language models (LLMs) Code synthesis is cheap, but far less accurate than directly processing each document with the LLM. We propose an extended code implementation, EVAPORATE-CODE+, which achieves better quality than direct extraction.
  arXiv  Detail & Related papers  (2023-04-19T06:00:26Z)
• Neural Methods for Logical Reasoning Over Knowledge Graphs [14.941769519278745]
  We focus on answering multi-hop logical queries on Knowledge Graphs (KGs) Most previous works have been unable to create models that accept full First-Order Logical (FOL) queries. We introduce a set of models that use Neural Networks to create one-point vector embeddings to answer the queries.
  arXiv  Detail & Related papers  (2022-09-28T23:10:09Z)
• On the Use of External Data for Spoken Named Entity Recognition [40.93448412171246]
  Recent advances in self-supervised speech representations have made it feasible to consider learning models with limited labeled data. We draw on a variety of approaches, including self-training, knowledge distillation, and transfer learning, and consider their applicability to both end-to-end models and pipeline approaches.
  arXiv  Detail & Related papers  (2021-12-14T18:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.