SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories

• URL: http://arxiv.org/abs/2409.07440v1
• Date: Wed, 11 Sep 2024 17:37:48 GMT
• Title: SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
• Authors: Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot,
• Abstract summary: Super aims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges, and 602 automatically generated problems for larger-scale development. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios.
• Score: 55.161075901665946
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPERaims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.

Related papers

• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)
• Learning Task Decomposition to Assist Humans in Competitive Programming [90.4846613669734]
  We introduce a novel objective for learning task decomposition, termed value (AssistV) We collect a dataset of human repair experiences on different decomposed solutions. Under 177 hours of human study, our method enables non-experts to solve 33.3% more problems, speeds them up by 3.3x, and empowers them to match unassisted experts.
  arXiv  Detail & Related papers  (2024-06-07T03:27:51Z)
• PECC: Problem Extraction and Coding Challenges [3.287942619833188]
  We introduce PECC, a novel benchmark derived from Advent Of Code (AoC) challenges and Project Euler. Unlike conventional benchmarks, PECC requires LLMs to interpret narrative-embedded problems, extract requirements, and generate code. Results show varying model performance between narrative and neutral problems, with specific challenges in the Euler math-based subset.
  arXiv  Detail & Related papers  (2024-04-29T15:02:14Z)
• Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision [98.97575836717931]
  Current AI alignment methodologies rely on human-provided demonstrations or judgments. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans?
  arXiv  Detail & Related papers  (2024-03-14T15:12:38Z)
• Effectiveness Assessment of Recent Large Vision-Language Models [78.69439393646554]
  This paper endeavors to evaluate the competency of popular large vision-language models (LVLMs) in specialized and general tasks. We employ six challenging tasks in three different application scenarios: natural, healthcare, and industrial. We examine the performance of three recent open-source LVLMs, including MiniGPT-v2, LLaVA-1.5, and Shikra, on both visual recognition and localization in these tasks.
  arXiv  Detail & Related papers  (2024-03-07T08:25:27Z)
• Competition-Level Problems are Effective LLM Evaluators [121.15880285283116]
  This paper aims to evaluate the reasoning capacities of large language models (LLMs) in solving recent programming problems in Codeforces. We first provide a comprehensive evaluation of GPT-4's peiceived zero-shot performance on this task, considering various aspects such as problems' release time, difficulties, and types of errors encountered. Surprisingly, theThoughtived performance of GPT-4 has experienced a cliff like decline in problems after September 2021 consistently across all the difficulties and types of problems.
  arXiv  Detail & Related papers  (2023-12-04T18:58:57Z)
• Assessing Quality-Diversity Neuro-Evolution Algorithms Performance in Hard Exploration Problems [10.871978893808533]
  Quality-Diversity (QD) methods are evolutionary algorithms inspired by nature's ability to produce high-performing niche organisms. In this paper, we highlight three candidate benchmarks exhibiting control problems in high dimension with exploration difficulties. We also provide open-source implementations in Jax allowing practitioners to run fast and numerous experiments on few compute resources.
  arXiv  Detail & Related papers  (2022-11-24T18:04:12Z)
• Measuring Progress on Scalable Oversight for Large Language Models [19.705153174673576]
  We present an experimental design centered on choosing tasks for which human specialists succeed but unaided humans and current general AI systems fail. We find that human participants who interact with an unreliable large-language-model dialog assistant through chat substantially outperform both the model alone and their own unaided performance.
  arXiv  Detail & Related papers  (2022-11-04T17:03:49Z)
• Learning to Solve Complex Tasks by Talking to Agents [39.08818632689814]
  Humans often solve complex problems by interacting with existing agents, such as AI assistants, that can solve simpler sub-tasks. Common NLP benchmarks aim for the development of self-sufficient models for every task. We propose a new benchmark called CommaQA that contains three kinds of complex reasoning tasks designed to be solved by talking'' to four agents with different capabilities.
  arXiv  Detail & Related papers  (2021-10-16T10:37:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.