The Weights can be Harmful: Pareto Search versus Weighted Search in Multi-Objective Search-Based Software Engineering

• URL: http://arxiv.org/abs/2202.03728v1
• Date: Tue, 8 Feb 2022 09:09:20 GMT
• Title: The Weights can be Harmful: Pareto Search versus Weighted Search in Multi-Objective Search-Based Software Engineering
• Authors: Tao Chen and Miqing Li
• Abstract summary: We show that the weights can, in fact, be harmful to the search process even in the presence of clear preferences. Our key finding is that weighted search reaches a certain level of solution quality by consuming relatively less resources at the early stage of the search.
• Score: 6.492599077364121
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In presence of multiple objectives to be optimized in Search-Based Software Engineering (SBSE), Pareto search has been commonly adopted. It searches for a good approximation of the problem's Pareto optimal solutions, from which the stakeholders choose the most preferred solution according to their preferences. However, when clear preferences of the stakeholders (e.g., a set of weights which reflect relative importance between objectives) are available prior to the search, weighted search is believed to be the first choice since it simplifies the search via converting the original multi-objective problem into a single-objective one and enable the search to focus on what only the stakeholders are interested in. This paper questions such a "weighted search first" belief. We show that the weights can, in fact, be harmful to the search process even in the presence of clear preferences. Specifically, we conduct a large scale empirical study which consists of 38 systems/projects from three representative SBSE problems, together with two types of search budget and nine sets of weights, leading to 604 cases of comparisons. Our key finding is that weighted search reaches a certain level of solution quality by consuming relatively less resources at the early stage of the search; however, Pareto search is at the majority of the time (up to 77% of the cases) significantly better than its weighted counterpart, as long as we allow a sufficient, but not unrealistic search budget. This, together with other findings and actionable suggestions in the paper, allows us to codify pragmatic and comprehensive guidance on choosing weighted and Pareto search for SBSE under the circumstance that clear preferences are available. All code and data can be accessed at: https://github.com/ideas-labo/pareto-vs-weight-for-sbse.

Related papers

• Swap Path Network for Robust Person Search Pre-training [0.0]
  We present the first framework for end-to-end person search pre-training. We show that our method is more effective, efficient, and robust for person search pre-training than recent backbone-only pre-training alternatives.
  arXiv  Detail & Related papers  (2024-12-06T21:35:26Z)
• Training Greedy Policy for Proposal Batch Selection in Expensive Multi-Objective Combinatorial Optimization [52.80408805368928]
  We introduce a novel greedy-style subset selection algorithm for batch acquisition. Our experiments on the red fluorescent proteins show that our proposed method achieves the baseline performance in 1.69x fewer queries.
  arXiv  Detail & Related papers  (2024-06-21T05:57:08Z)
• A Visual Active Search Framework for Geospatial Exploration [36.31732056074638]
  Many problems can be viewed as forms of geospatial search aided by aerial imagery. We model this class of problems in a visual active search (VAS) framework, which has three key inputs. We propose a reinforcement learning approach for VAS that learns a meta-search policy from a collection of fully annotated search tasks.
  arXiv  Detail & Related papers  (2022-11-28T21:53:05Z)
• MICO: Selective Search with Mutual Information Co-training [14.456028769565386]
  MICO is a Mutual Information CO-training framework for selective search. After training, MICO does not only cluster the documents, but also routes unseen queries to the relevant clusters for efficient retrieval.
  arXiv  Detail & Related papers  (2022-09-09T16:26:52Z)
• Multidimensional Assignment Problem for multipartite entity resolution [69.48568967931608]
  Multipartite entity resolution aims at integrating records from multiple datasets into one entity. We apply two procedures, a Greedy algorithm and a large scale neighborhood search, to solve the assignment problem. We find evidence that design-based multi-start can be more efficient as the size of databases grow large.
  arXiv  Detail & Related papers  (2021-12-06T20:34:55Z)
• Compositional Attention: Disentangling Search and Retrieval [66.7108739597771]
  Multi-head, key-value attention is the backbone of the Transformer model and its variants. Standard attention heads learn a rigid mapping between search and retrieval. We propose a novel attention mechanism, called Compositional Attention, that replaces the standard head structure.
  arXiv  Detail & Related papers  (2021-10-18T15:47:38Z)
• Exposing Query Identification for Search Transparency [69.06545074617685]
  We explore the feasibility of approximate exposing query identification (EQI) as a retrieval task by reversing the role of queries and documents in two classes of search systems. We derive an evaluation metric to measure the quality of a ranking of exposing queries, as well as conducting an empirical analysis focusing on various practical aspects of approximate EQI.
  arXiv  Detail & Related papers  (2021-10-14T20:19:27Z)
• Efficient Active Search for Combinatorial Optimization Problems [1.6543719822033436]
  We show that (efficient) active search enables learned models to effectively solve instances that are much larger than those seen during training. The proposed methods offer a simple way to significantly improve the search performance of a given model and outperform state-of-the-art machine learning based methods on routing problems.
  arXiv  Detail & Related papers  (2021-06-09T15:08:03Z)
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking [97.60915598958968]
  We propose a one-shot neural ensemble architecture search (NEAS) solution that addresses the two challenges. For the first challenge, we introduce a novel diversity-based metric to guide search space shrinking. For the second challenge, we enable a new search dimension to learn layer sharing among different models for efficiency purposes.
  arXiv  Detail & Related papers  (2021-04-01T16:29:49Z)
• Robust Partial Matching for Person Search in the Wild [70.6661871706788]
  This paper proposes an Align-to-Part Network (APNet) for person detection and re-Identification (reID) APNet refines detected bounding boxes to cover the estimated holistic body regions. It achieves competitive performance on existing person search benchmarks like CUHK-SYSU and PRW.
  arXiv  Detail & Related papers  (2020-04-20T14:21:03Z)
• How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance [9.040916182677963]
  This paper reviews studies on quality evaluation for multi-objective optimization in Search-Based SE. We conduct an in-depth analysis of quality evaluation indicators/methods and general situations in SBSE. We codify a methodological guidance for selecting and using evaluation methods in different SBSE scenarios.
  arXiv  Detail & Related papers  (2020-02-20T22:12:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.