AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access

• URL: http://arxiv.org/abs/2603.22376v1
• Date: Mon, 23 Mar 2026 09:29:08 GMT
• Title: AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access
• Abstract summary: We present an AI Co-Scientist framework that automates the full search ranking research pipeline.<n>This is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research.
• Score: 60.37646376282544
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in AI agents for software engineering and scientific discovery have demonstrated remarkable capabilities, yet their application to developing novel ranking models in commercial search engines remains unexplored. In this paper, we present an AI Co-Scientist framework that automates the full search ranking research pipeline: from idea generation to code implementation and GPU training job scheduling with expert in the loop. Our approach strategically employs single-LLM agents for routine tasks while leveraging multi-LLM consensus agents (GPT 5.2, Gemini Pro 3, and Claude Opus 4.5) for challenging phases such as results analysis and idea generation. To our knowledge, this is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research. We demonstrate that this framework discovered a novel technique for handling sequence features, with all model enhancements produced automatically, yielding substantial offline performance improvements. Our findings suggest that AI systems can discover ranking architectures comparable to those developed by human experts while significantly reducing routine research workloads.

Related papers

• AIBuildAI-2: A Knowledge-Enhanced Agent for Automatically Building AI Models [31.05147308924019]
  AI models underpin data-centric applications from image and text processing to scientific discovery in biology, physics, and chemistry.<n>Yet developing them remains heavily manual, requiring practitioners to design architectures, build training pipelines, and iteratively refine solutions.<n>We introduce AIBuildAI-2, a knowledge-enhanced agent with an external, evolving knowledge system for automatically building AI models.
  arXiv  Detail & Related papers  (2026-05-27T02:44:13Z)
• AIBuildAI: An AI Agent for Automatically Building AI Models [31.05147308924019]
  We introduce AIBuildAI, an AI agent that automatically builds AI models from a task description and training data.<n>We evaluate AIBuildAI on MLE-Bench, a benchmark of realistic Kaggle-style AI development tasks.
  arXiv  Detail & Related papers  (2026-04-15T22:17:05Z)
• AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery [49.77172027295716]
  AutoSOTA is an end-to-end automated research system that advances the latest State-Of-The-Art (SOTA) models published in top-tier AI papers.<n>It successfully discovers 105 new SOTA models that surpass the original reported methods, averaging approximately five hours per paper.
  arXiv  Detail & Related papers  (2026-04-07T07:52:01Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• The AI Roles Continuum: Blurring the Boundary Between Research and Engineering [0.0]
  We propose a framework in which Research Scientists, Research Engineers, Applied Scientists, and Machine Learning Engineers occupy overlapping positions rather than discrete categories.<n>We show that core competencies such as distributed systems design, large-scale training and optimization, rigorous experimentation, and publication-minded inquiry are now broadly shared across titles.<n>Treating roles as fluid rather than siloed shortens research-to-production loops, improves iteration velocity, and strengthens organizational learning.
  arXiv  Detail & Related papers  (2025-12-31T05:00:58Z)
• Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training [67.895981259683]
  General AI Agents are increasingly recognized as foundational frameworks for the next generation of artificial intelligence.<n>Current agent systems are either closed-source or heavily reliant on a variety of paid APIs and proprietary tools.<n>We present Cognitive Kernel-Pro, a fully open-source and (to the maximum extent) free multi-module agent framework.
  arXiv  Detail & Related papers  (2025-08-01T08:11:31Z)
• AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench [76.23945148787051]
  We formalize AI research agents as search policies that navigate a space of candidate solutions, iteratively modifying them using operators.<n>Our best pairing of search strategy and operator set achieves a state-of-the-art result on MLE-bench lite, increasing the success rate of achieving a Kaggle medal from 39.6% to 47.7%.
  arXiv  Detail & Related papers  (2025-07-03T11:59:15Z)
• MLGym: A New Framework and Benchmark for Advancing AI Research Agents [51.9387884953294]
  We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing large language models on AI research tasks.<n>This is the first Gym environment for machine learning (ML) tasks, enabling research on reinforcement learning (RL) algorithms for training such agents.<n>We evaluate a number of frontier large language models (LLMs) on our benchmarks such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5 Pro.
  arXiv  Detail & Related papers  (2025-02-20T12:28:23Z)
• ML Research Benchmark [0.0]
  We present the ML Research Benchmark (MLRB), comprising 7 competition-level tasks derived from recent machine learning conference tracks. This paper introduces a novel benchmark and evaluates it using agent scaffolds powered by frontier models, including Claude-3 and GPT-4o. The results indicate that the Claude-3.5 Sonnet agent performs best across our benchmark, excelling in planning and developing machine learning models.
  arXiv  Detail & Related papers  (2024-10-29T21:38:42Z)
• Automated Design of Agentic Systems [5.404186221463082]
  We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs.<n>We show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents.
  arXiv  Detail & Related papers  (2024-08-15T21:59:23Z)
• System for systematic literature review using multiple AI agents: Concept and an empirical evaluation [3.453564255183234]
  Systematic literature review ( SLR) is foundational to evidence-based research.<n>We introduce a novel multi-AI-agent system designed to fully automate SLRs.<n>We show how it substantially reduces the time and effort traditionally required for SLRs while maintaining comprehensiveness and precision.
  arXiv  Detail & Related papers  (2024-03-13T10:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.