Why is the winner the best?

• URL: http://arxiv.org/abs/2303.17719v1
• Date: Thu, 30 Mar 2023 21:41:42 GMT
• Title: Why is the winner the best?
• Authors: Matthias Eisenmann, Annika Reinke, Vivienn Weru, Minu Dietlinde Tizabi, Fabian Isensee, Tim J. Adler, Sharib Ali, Vincent Andrearczyk, Marc Aubreville, Ujjwal Baid, Spyridon Bakas, Niranjan Balu, Sophia Bano, Jorge Bernal, Sebastian Bodenstedt, Alessandro Casella, Veronika Cheplygina, Marie Daum, Marleen de Bruijne, Adrien Depeursinge, Reuben Dorent, Jan Egger, David G. Ellis, Sandy Engelhardt, Melanie Ganz, Noha Ghatwary, Gabriel Girard, Patrick Godau, Anubha Gupta, Lasse Hansen, Kanako Harada, Mattias Heinrich, Nicholas Heller, Alessa Hering, Arnaud Huaulm\'e, Pierre Jannin, Ali Emre Kavur, Old\v{r}ich Kodym, Michal Kozubek, Jianning Li, Hongwei Li, Jun Ma, Carlos Mart\'in-Isla, Bjoern Menze, Alison Noble, Valentin Oreiller, Nicolas Padoy, Sarthak Pati, Kelly Payette, Tim R\"adsch, Jonathan Rafael-Pati\~no, Vivek Singh Bawa, Stefanie Speidel, Carole H. Sudre, Kimberlin van Wijnen, Martin Wagner, Donglai Wei, Amine Yamlahi, Moi Hoon Yap, Chun Yuan, Maximilian Zenk, Aneeq Zia, David Zimmerer, Dogu Baran Aydogan, Binod Bhattarai, Louise Bloch, Raphael Br\"ungel, Jihoon Cho, Chanyeol Choi, Qi Dou, Ivan Ezhov, Christoph M. Friedrich, Clifton Fuller, Rebati Raman Gaire, Adrian Galdran, \'Alvaro Garc\'ia Faura, Maria Grammatikopoulou, SeulGi Hong, Mostafa Jahanifar, Ikbeom Jang, Abdolrahim Kadkhodamohammadi, Inha Kang, Florian Kofler, Satoshi Kondo, Hugo Kuijf, Mingxing Li, Minh Huan Luu, Toma\v{z} Martin\v{c}i\v{c}, Pedro Morais, Mohamed A. Naser, Bruno Oliveira, David Owen, Subeen Pang, Jinah Park, Sung-Hong Park, Szymon P{\l}otka, Elodie Puybareau, Nasir Rajpoot, Kanghyun Ryu, Numan Saeed, Adam Shephard, Pengcheng Shi, Dejan \v{S}tepec, Ronast Subedi, Guillaume Tochon, Helena R. Torres, Helene Urien, Jo\~ao L. Vila\c{c}a, Kareem Abdul Wahid, Haojie Wang, Jiacheng Wang, Liansheng Wang, Xiyue Wang, Benedikt Wiestler, Marek Wodzinski, Fangfang Xia, Juanying Xie, Zhiwei Xiong, Sen Yang, Yanwu Yang, Zixuan Zhao, Klaus Maier-Hein, Paul F. J\"ager, Annette Kopp-Schneider, and Lena Maier-Hein
• Abstract summary: We performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE I SBI 2021 and MICCAI 2021. Winning solutions typically include the use of multi-task learning (63%), and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%) Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases.
• Score: 78.74409216961632
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

Related papers

• Approaches to the Algorithmic Allocation of Public Resources: A Cross-disciplinary Review [0.0]
  We performed a cross disciplinary literature review to understand approaches taken for different areas of algorithmic allocation. We analyzed the 75 papers from the lenses of optimization goals, techniques, interpretability, flexibility, bias, ethical considerations, and performance. We found considerable potential for performance gains, with optimization techniques often decreasing waiting times and increasing success rate by as much as 50%.
  arXiv  Detail & Related papers  (2023-10-10T09:46:01Z)
• Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO [50.58083807719749]
  We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.
  arXiv  Detail & Related papers  (2023-08-30T07:16:11Z)
• A Gold Standard Dataset for the Reviewer Assignment Problem [117.59690218507565]
  "Similarity score" is a numerical estimate of the expertise of a reviewer in reviewing a paper. Our dataset consists of 477 self-reported expertise scores provided by 58 researchers. For the task of ordering two papers in terms of their relevance for a reviewer, the error rates range from 12%-30% in easy cases to 36%-43% in hard cases.
  arXiv  Detail & Related papers  (2023-03-23T16:15:03Z)
• Biomedical image analysis competitions: The state of current participation practice [143.52578599912326]
  We designed a survey to shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis. The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures.
  arXiv  Detail & Related papers  (2022-12-16T16:44:46Z)
• Retrospective on the 2021 BASALT Competition on Learning from Human Feedback [92.37243979045817]
  The goal of the competition was to promote research towards agents that use learning from human feedback (LfHF) techniques to solve open-world tasks. Rather than mandating the use of LfHF techniques, we described four tasks in natural language to be accomplished in the video game Minecraft. Teams developed a diverse range of LfHF algorithms across a variety of possible human feedback types.
  arXiv  Detail & Related papers  (2022-04-14T17:24:54Z)
• The First AI4TSP Competition: Learning to Solve Stochastic Routing Problems [10.388013100067266]
  This paper reports on the first international competition on AI for the traveling salesman problem (TTSP) at the 2021 International Conference on Artificial Intelligence (IJCAI-21) The competition asked the participants to develop algorithms to solve a time-dependent orienteering problem with weights and time windows (TD-OPSWTW) The winning methods described in this work have advanced the state-of-the-art in AI for routing problems using AI.
  arXiv  Detail & Related papers  (2022-01-25T16:55:33Z)
• Lung-Originated Tumor Segmentation from Computed Tomography Scan (LOTUS) Benchmark [48.30502612686276]
  Lung cancer is one of the deadliest cancers, and its effective diagnosis and treatment depend on the accurate delineation of the tumor. Human-centered segmentation, which is currently the most common approach, is subject to inter-observer variability. The 2018 VIP Cup started with a global engagement from 42 countries to access the competition data. In a nutshell, all the algorithms proposed during the competition, are based on deep learning models combined with a false positive reduction technique.
  arXiv  Detail & Related papers  (2022-01-03T03:06:38Z)
• CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions [25.791936837340877]
  The first Continual Learning in Computer Vision challenge held at CVPR in 2020 has been one of the first opportunities to evaluate different continual learning algorithms. We report the main results of the competition, which counted more than 79 teams registered, 11 finalists and 2300$ in prizes.
  arXiv  Detail & Related papers  (2020-09-14T08:53:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.