From Brittle to Robust: Improving LLM Annotations for SE Optimization

• URL: http://arxiv.org/abs/2603.22474v1
• Date: Mon, 23 Mar 2026 18:42:15 GMT
• Title: From Brittle to Robust: Improving LLM Annotations for SE Optimization
• Abstract summary: When human expertise is scarce, SE researchers sometimes ask large language models (LLMs) for the missing labels.<n>To address this task, we propose a novel LLM prompting strategy called SynthCore.<n>When one opinion fails, SynthCore's combines multiple separated opinions generated by LLMs into an ensemble of few-shot learners.
• Score: 4.082216579462797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software analytics often builds from labeled data. Labeling can be slow, error prone, and expensive. When human expertise is scarce, SE researchers sometimes ask large language models (LLMs) for the missing labels. While this has been successful in some domains, recent results show that LLM-based labeling has blind spots. Specifically, their labeling is not effective for higher dimensional multi-objective problems. To address this task, we propose a novel LLM prompting strategy called SynthCore. When one opinion fails, SynthCore's combines multiple separated opinions generated by LLMs (with no knowledge of each others' answers) into an ensemble of few-shot learners. Simpler than other strategies (e.g. chain-of-thought, multi-agent-debate, etc) SynthCore aggregates results from multiple single prompt sessions (with no crossover between them). SynthCore has been tested on 49 SE multi-objective optimization tasks, handling tasks as diverse as software project management, Makefile configuration, and hyperparameter optimization. SynthCore's ensemble found optimizations that are better than state-of-the-art alternative approaches (Gaussian Process Models, Tree of Parzen Estimators, active learners in both exploration and exploitation mode). Importantly, these optimizations were made using data labeled by LLMs, without any human opinions. From these experiments, we conclude that ensembles of few shot learners can successfully annotate high dimensional multi-objective tasks. Further, we speculate that other successful few-shot prompting results could be quickly and easily enhanced using SynthCore's ensemble approach. To support open science, all our data and scripts are available at https://github.com/lohithsowmiyan/lazy-llm/tree/clusters.

Related papers

• Can LLMs Build a MaxSAT Solver from Papers? The CoreForge Experience [5.812499828391904]
  We report on an experience in using large language models (LLMs) to build an unweighted MaxSAT solver from research papers.<n>The project focuses on unsatisfiability-based MaxSAT algorithms and follows an iterative workflow that combines paper discussions with ChatGPT.<n>We summarize what worked, what remained difficult, and the lessons for future LLM-assisted solver development.
  arXiv  Detail & Related papers  (2026-07-16T10:37:02Z)
• Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration [57.23877089929136]
  Graph-of-Agents (GoA) is a new graph-based framework for modeling multi-agent LLM communication.<n>GoA achieves superior performance using only 3 selected agents, outperforming recent multi-agent LLM baselines.
  arXiv  Detail & Related papers  (2026-04-18T21:13:03Z)
• How Low Can You Go? The Data-Light SE Challenge [4.282746516699565]
  Much of software engineering (SE) research assumes that progress depends on massive datasets and CPU-intensives.<n>Counter-evidence presented in this paper suggests otherwise, including software configuration and performance tuning, cloud and systems optimization, project and process-level decision modeling, behavioral analytics, financial risk modeling, project health prediction, reinforcement learning tasks, sales forecasting, and software testing.<n>Our results highlight that some SE tasks may be better served by lightweight approaches that demand fewer labels and far less computation.
  arXiv  Detail & Related papers  (2025-12-15T16:49:50Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Teamwork makes the dream work: LLMs-Based Agents for GitHub README.MD Summarization [7.330697128881243]
  We propose Metagente as a novel approach to amplify the synergy of various Large Language Models (LLMs)<n>Metagente is a Multi-Agent framework based on a series of LLMs to self-optimize the system through evaluation, feedback, and cooperation among specialized agents.<n>The performance gain compared to GitSum, the most relevant benchmark, ranges from 27.63% to 60.43%.
  arXiv  Detail & Related papers  (2025-03-13T20:42:39Z)
• Online Prompt Selection for Program Synthesis [14.188804919928591]
  Large Language Models (LLMs) demonstrate impressive capabilities in the domain of program synthesis.<n>A key challenge for the user is to identify when an LLM is the right choice of solver, and the appropriate LLM to call for a given synthesis task, but also the right way to call it.<n>We use a multi-armed bandit algorithm to select which symbolic solver, or LLM and prompt combination to deploy in order to maximize a given reward function.
  arXiv  Detail & Related papers  (2025-01-09T13:57:09Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• Aligning Language Models with Demonstrated Feedback [58.834937450242975]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors.<n>We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration [68.29746557968107]
  We propose a novel framework for multi-agent collaboration that introduces Reinforced Advantage feedback (ReAd) for efficient self-refinement of plans.<n> Experiments on Over-AI and a difficult variant of RoCoBench show that ReAd surpasses baselines in success rate, and also significantly decreases the interaction steps of agents.
  arXiv  Detail & Related papers  (2024-05-23T08:33:19Z)
• SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation [55.2480439325792]
  We study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance.
  arXiv  Detail & Related papers  (2024-05-16T12:22:41Z)
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation [96.71370747681078]
  We introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate.
  arXiv  Detail & Related papers  (2023-10-05T04:06:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.