Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder

• URL: http://arxiv.org/abs/2608.19957v1
• Date: Thu, 20 Aug 2026 12:24:53 GMT
• Title: Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder
• Abstract summary: We present an open benchmark of 3,413 real-world, PII-scrubbed query-code pairs.<n>We fine-tune on 784,057 synthetic triplets generated by google/gemma-4-26B-A4B-it from public code repositories.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural language code retrieval is a rapidly evolving task in computer science. However, the 1C:Enterprise ecosystem combines Russian syntax with highly domain-specific terminology, for which open datasets and specialized models have been virtually non-existent. We present a comprehensive pipeline for 1C code retrieval: an open benchmark of 3,413 real-world, PII-scrubbed query-code pairs, a reproducible evaluation harness, and a specialized bi-encoder. To overcome scarce labeled data, we fine-tune on 784,057 synthetic triplets generated by google/gemma-4-26B-A4B-it from public code repositories, using Matryoshka Representation Learning (MRL) and a privacy-aware tokenizer. Because the benchmark subsets differ in size, we report balanced-subset macro, query-weighted micro, and forum-only results. Our model reaches 0.5992 balanced macro nDCG@10, 0.5044 micro, and 0.4617 on forum, versus 0.4932 macro for the baseline architecture and 0.5404 for google/embeddinggemma-300m. Removing every benchmark example flagged by the conservative exact/13-gram overlap audit leaves 0.6011 balanced macro (0.5010 micro), indicating that detected train-benchmark overlap does not explain the headline result. MRL truncation to 256 dimensions preserves 99.9% of retrieval quality while reducing dense-index storage and exact similarity arithmetic by a factor of three.

Related papers

• SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction [12.71956248550879]
  LLM agents can generate reproduction code, yet often produce scientifically unfaithful implementations.<n>We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications.<n>We introduce SemanticAlign-Bench, a diagnostic benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025.
  arXiv  Detail & Related papers  (2026-08-25T08:48:48Z)
• Consensus-gated Multi-Agent Neural Architecture Search for Seismic Fault Segmentation [0.17188280334580194]
  Neural networks for seismic fault segmentation are often borrowed from computer vision and medical imaging domains.<n>Classical neural architecture search (NAS) is restricted to hand-crafted search spaces and large compute budgets.<n>We present an agentic NAS system in which a panel of three large language models (Claude, GPT-5.1, and Gemini2.5Pro) debates each candidate architecture to unanimous consensus.
  arXiv  Detail & Related papers  (2026-08-14T02:41:37Z)
• MediaWiki Code2Code Search: Neural Retrieval for the Semantic Discovery of Open-Source Software Entities [0.5279475826661643]
  We present Code2 InformationCode Search, a neural retrieval system for semantic code-to-code discovery.<n>By 1.29 million structural (functions, types, and templates) Code across 2,500+ large repositories, our system enables retrieval based on computational intent.<n>We employ a split-build architecture, decouplingintensive offline indexing from a CPU-only serving layer.
  arXiv  Detail & Related papers  (2026-07-29T11:07:07Z)
• ScoreGate: Adaptive Chunk Selection for Retrieval-Augmented Generation via Dual-Score Statistical Fusion [0.0]
  We describe ScoreGate, a lightweight score-space decision mechanism that controls retrieval cardinality at inference time.<n>On MS MARCO (200 dev queries), ScoreGate achieves MRR@10 = 0.401 with 35% fewer retained chunks than Standard Top-K.<n>Results on both MS MARCO and real-world production traffic suggest that adaptive retrieval cardinality can improve retrieval efficiency without degrading retrieval quality.
  arXiv  Detail & Related papers  (2026-06-12T08:51:24Z)
• Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages [1.2891210250935148]
  We build a two-stage retriever (bi-encoder followed by cross-encoder reranker) on Gemini-generated synthetic data covering English, Spanish, Catalan, Italian, Portuguese and French.<n>The bi-encoder alone matches BioBERT-ST on MRR (0.876 vs. 0.866) and overtakes it on R@3 (0.650 vs. 0.626) and R@5 (0.804 vs. 0.790) without English biomedical pretraining.<n> Adding a cross-encoder reranker lifts aggregate R@5 to 0.822 and dominates on four of five languages (+0.0
  arXiv  Detail & Related papers  (2026-05-28T20:06:43Z)
• SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers [55.39407031861402]
  We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
  arXiv  Detail & Related papers  (2026-05-27T20:08:47Z)
• CLARC: C/C++ Benchmark for Robust Code Search [2.225731679677886]
  We present CLARC, a C/C++ benchmark built from real-world GitHub repositories.<n>Clarc contains 1,245 query-code pairs for evaluation and 5,472 pairs for training.
  arXiv  Detail & Related papers  (2026-03-04T18:57:37Z)
• MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm [60.14048367611333]
  MonkeyOCR is a vision-language model for document parsing.<n>It advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm.
  arXiv  Detail & Related papers  (2025-06-05T16:34:57Z)
• Let the Code LLM Edit Itself When You Edit the Code [50.46536185784169]
  underlinetextbfPositional textbfIntegrity textbfEncoding (PIE)<n>PIE reduces computational overhead by over 85% compared to the standard full recomputation approach.<n>Results demonstrate that PIE reduces computational overhead by over 85% compared to the standard full recomputation approach.
  arXiv  Detail & Related papers  (2024-07-03T14:34:03Z)
• Linear-time Minimum Bayes Risk Decoding with Reference Aggregation [52.1701152610258]
  Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations. It requires the pairwise calculation of a utility metric, which has quadratic complexity. We propose to approximate pairwise metric scores with scores calculated against aggregated reference representations.
  arXiv  Detail & Related papers  (2024-02-06T18:59:30Z)
• Tackling Long Code Search with Splitting, Encoding, and Aggregating [67.02322603435628]
  We propose a new baseline SEA (Split, Encode and Aggregate) for long code search. It splits long code into code blocks, encodes these blocks into embeddings, and aggregates them to obtain a comprehensive long code representation. With GraphCodeBERT as the encoder, SEA achieves an overall mean reciprocal ranking score of 0.785, which is 10.1% higher than GraphCodeBERT on the CodeSearchNet benchmark.
  arXiv  Detail & Related papers  (2022-08-24T02:27:30Z)
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering [63.92224685262063]
  CoSQA dataset includes 20,604 labels for pairs of natural language queries and codes. We introduce a contrastive learning method dubbed CoCLR to enhance query-code matching. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%.
  arXiv  Detail & Related papers  (2021-05-27T15:37:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.