Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions

• URL: http://arxiv.org/abs/2604.27763v1
• Date: Thu, 30 Apr 2026 11:52:50 GMT
• Title: Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions
• Abstract summary: textscIntent2Tx grounds natural language intents in real-world protocol interactions across 11 categories.<n>textscIntent2Tx serves as a critical foundation for developing autonomous, reliable agents in intent-centric Web3 ecosystems.
• Score: 6.606052122056915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The emergence of Large Language Models (LLMs) offers a transformative interface for Web3, yet existing benchmarks fail to capture the complexity of translating high-level user intents into functionally correct, state-dependent on-chain transactions. We present \textsc{Intent2Tx}, a high-fidelity benchmark featuring 29,921 single-step and 1,575 multi-step instances meticulously derived from 300 days of real-world Ethereum mainnet traces. Unlike prior works that rely on synthetic instructions, \textsc{Intent2Tx} grounds natural language intents in real-world protocol interactions across 11 categories, including diverse long-tail Decentralized Finance (DeFi) primitives. To enable rigorous evaluation, we propose an execution-aware framework that transcends surface-level text matching by employing differential state analysis on forked mainnet environments. Our extensive evaluation of 16 state-of-the-art LLMs reveals that while scaling and retrieval-augmentation enhance logical consistency and parameter precision, current models struggle with out-of-distribution generalization and multi-step planning. Crucially, our execution-based analysis demonstrates that syntactically valid outputs often fail to achieve intended state transitions, highlighting a significant gap in current "reasoning-to-execution" capabilities. \textsc{Intent2Tx} serves as a critical foundation for developing autonomous, reliable agents in intent-centric Web3 ecosystems. Code and data: https://anonymous.4open.science/r/Intent2Tx_Bench-97FF .

Related papers

• The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis [50.473217152006875]
  MIMIC is a framework that leverages executable code as a rigorous medium for reasoning data synthesis.<n>MIMIC transforms algorithms into verifiable reasoning trajectories through narrative fusion, code-guided test synthesis, and dynamic code instrumentation.<n>Our method significantly elevates accuracy across general reasoning, complex mathematical benchmarks, and fine-grained deterministic tasks.
  arXiv  Detail & Related papers  (2026-09-13T17:04:16Z)
• $\mathcal{S}^2$IT: Stepwise Syntax Integration Tuning for Large Language Models in Aspect Sentiment Quad Prediction [51.165432266846096]
  syntactic structure information is underutilized in the generative paradigm of large language models (LLMs)<n>S2IT progressively integrates syntactic structure knowledge into LLMs through a multi-step tuning process.<n> Experiments demonstrate that S2IT significantly improves state-of-the-art performance across multiple datasets.
  arXiv  Detail & Related papers  (2026-04-25T13:24:29Z)
• TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications [2.2508462342902633]
  Large language models (LLMs) are increasingly deployed in the telecommunications domain for critical tasks.<n>We introduce TeleEmbedBench, the first large-scale, multi-corpus embedding benchmark designed specifically for telecommunications.
  arXiv  Detail & Related papers  (2026-04-20T04:00:13Z)
• OMNIINTENT: A Trusted Intent-Centric Framework for User-Friendly Web3 [6.606052122056915]
  Intent-Centric finance allows users to focus on their trading intents rather than the underlying execution details.<n>We present OMNIINTENT, a language-runtime co-design that reconciles these requirements.<n>Our full-stack prototype processes diverse DeFi scenarios, achieving 89.6% throughput coverage, up to 7.3x speedup via parallel execution, and feasibility-prediction accuracy up to 99.2% with low latency.
  arXiv  Detail & Related papers  (2026-03-04T15:24:47Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• Explore-Execute Chain: Towards an Efficient Structured Reasoning Paradigm [8.405729585427226]
  Chain-of-Thought (CoT) and its variants have markedly advanced the reasoning abilities of Large Language Models (LLMs)<n>We propose the Explore-Execute Chain ($E2C$), a structured reasoning framework that decouples reasoning into two distinct phases.
  arXiv  Detail & Related papers  (2025-09-28T15:48:40Z)
• Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes [33.80591142965565]
  We present CODE2BENCH, a pipeline for dynamically constructing robust and contamination-resistant benchmarks from real-world GitHub repositories.<n>Specifically, CODE2BENCH introduces three key innovations: (1) Automated Dynamism, achieved through periodic ingestion of recent code to minimize training data contamination; (2) Scope Graph-based dependency analysis, which enables structured classification of functions into benchmark instances with controlled dependency levels; and (3) Property-Based Testing (PBT) for the automated synthesis of rigorous test suites.
  arXiv  Detail & Related papers  (2025-08-10T05:06:36Z)
• Evaluating LLMs on Sequential API Call Through Automated Test Generation [10.621357661774244]
  StateGen is an automated framework designed to generate diverse coding tasks involving sequential API interactions.<n>We construct StateEval, a benchmark encompassing 120 verified test cases spanning across three representative scenarios.<n> Experimental results confirm that StateGen can effectively generate challenging and realistic API-oriented tasks.
  arXiv  Detail & Related papers  (2025-07-13T03:52:51Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• FSD V2: Improving Fully Sparse 3D Object Detection with Virtual Voxels [57.05834683261658]
  We present FSDv2, an evolution that aims to simplify the previous FSDv1 while eliminating the inductive bias introduced by its handcrafted instance-level representation. We develop a suite of components to complement the virtual voxel concept, including a virtual voxel encoder, a virtual voxel mixer, and a virtual voxel assignment strategy.
  arXiv  Detail & Related papers  (2023-08-07T17:59:48Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.