Bridging the Evaluation Gap: Standardized Benchmarks for Multi-Objective Search

• URL: http://arxiv.org/abs/2603.24084v1
• Date: Wed, 25 Mar 2026 08:45:33 GMT
• Title: Bridging the Evaluation Gap: Standardized Benchmarks for Multi-Objective Search
• Abstract summary: We introduce the first comprehensive, standardized benchmark suite for exact and approximate multi-objective search (MOS)<n>Our suite spans four structurally diverse domains: real-world road networks, structured synthetic graphs, game-based grid environments, and high-dimensional robotic motion-planning roadmaps.
• Score: 19.834978083336022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Empirical evaluation in multi-objective search (MOS) has historically suffered from fragmentation, relying on heterogeneous problem instances with incompatible objective definitions that make cross-study comparisons difficult. This standardization gap is further exacerbated by the realization that DIMACS road networks, a historical default benchmark for the field, exhibit highly correlated objectives that fail to capture diverse Pareto-front structures. To address this, we introduce the first comprehensive, standardized benchmark suite for exact and approximate MOS. Our suite spans four structurally diverse domains: real-world road networks, structured synthetic graphs, game-based grid environments, and high-dimensional robotic motion-planning roadmaps. By providing fixed graph instances, standardized start-goal queries, and both exact and approximate reference Pareto-optimal solution sets, this suite captures a full spectrum of objective interactions: from strongly correlated to strictly independent. Ultimately, this benchmark provides a common foundation to ensure future MOS evaluations are robust, reproducible, and structurally comprehensive.

Related papers

• Structure Aware Neural Architecture Search for Mixture of Experts [3.100317701802537]
  We propose an architecture search framework that makes the alignment between experts and the structure of the data emerge as a search variable.<n>We show that it coincides with the incomplete-data maximum likelihood of a latent-variable mixture, and solve it by a generalised Expectation-Maximisation procedure.<n>We prove that the iterates converge whenever the surrogate errors are summable, and that at every limit point no candidate the search produces improves the true objective.
  arXiv  Detail & Related papers  (2026-08-30T14:37:23Z)
• Cross-View Feature Matching: Survey, Benchmarking, and Foundation-Model Perspectives [74.46417373478194]
  Cross-view feature matching aims to establish reliable correspondences across images with large viewpoint variations.<n>Over the past decade, the field has evolved from task-specific models toward increasingly unified and generalizable correspondence models.<n>Despite these advances, existing studies remain highly diverse in their problem formulations, model architectures, training paradigms, and evaluation protocols.
  arXiv  Detail & Related papers  (2026-08-11T16:00:40Z)
• Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation [47.29900218955283]
  Mutual information estimation is a central problem in machine learning and statistics.<n>Existing benchmarks evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored.<n>We address this gap with a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective.
  arXiv  Detail & Related papers  (2026-07-03T16:55:55Z)
• HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding [1.8482679687103294]
  HAVEN is a hierarchically aligned multimodal benchmark for unified video understanding.<n>Built upon this unified annotation paradigm, we propose a comprehensive evaluation suite spanning summarization, temporal reasoning, multimodal grounding, and saliency ranking.
  arXiv  Detail & Related papers  (2026-05-19T00:48:14Z)
• MOSIV: Multi-Object System Identification from Videos [54.53609234233857]
  We propose a new framework that directly optimize for continuous, per-object material parameters using a differentiable simulator guided by geometric objectives derived from video.<n>We also present a new synthetic benchmark with contact-rich, multi-object interactions to facilitate evaluation.
  arXiv  Detail & Related papers  (2026-03-06T08:16:16Z)
• Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities [2.9203730377983654]
  Existing benchmarks fail to reflect real-world use or isolate compliance from task success.<n>We introduce MOSAIC, a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints.<n>We show that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position.
  arXiv  Detail & Related papers  (2026-01-26T15:02:15Z)
• Procrustean Bed for AI-Driven Retrosynthesis: A Unified Framework for Reproducible Evaluation [0.0]
  RetroCast is a unified evaluation suite that standardizes heterogeneous model outputs into a common schema.<n>We evaluate leading search-based and sequence-based algorithms on a new suite of standardized benchmarks.
  arXiv  Detail & Related papers  (2025-12-08T01:26:39Z)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning [6.686583184622338]
  We formalize the concept of generalization in Multi-Objective Reinforcement Learning (MORL) and how it can be evaluated.<n>We contribute a novel benchmark featuring diverse multi-objective domains with parameterized environment configurations.<n>Our baseline evaluations of state-of-the-art MORL algorithms on this benchmark reveals limited generalization capabilities, suggesting significant room for improvement.
  arXiv  Detail & Related papers  (2025-03-02T08:50:14Z)
• GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models [56.63218531256961]
  We introduce GenBench, a benchmarking suite specifically tailored for evaluating the efficacy of Genomic Foundation Models. GenBench offers a modular and expandable framework that encapsulates a variety of state-of-the-art methodologies. We provide a nuanced analysis of the interplay between model architecture and dataset characteristics on task-specific performance.
  arXiv  Detail & Related papers  (2024-06-01T08:01:05Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• Relational Proxies: Emergent Relationships as Fine-Grained Discriminators [52.17542855760418]
  We propose a novel approach that leverages information between the global and local part of an object for encoding its label. We design Proxies based on our theoretical findings and evaluate it on seven challenging fine-grained benchmark datasets. We also experimentally validate our theory and obtain consistent results across multiple benchmarks.
  arXiv  Detail & Related papers  (2022-10-05T11:08:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.