The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Results

• URL: http://arxiv.org/abs/2109.00498v1
• Date: Tue, 31 Aug 2021 01:29:56 GMT
• Title: The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Results
• Authors: Stanley Bak, Changliu Liu, Taylor Johnson
• Abstract summary: This report summarizes the second International Verification of Neural Networks Competition (VNN-COMP 2021) The goal of the competition is to provide an objective comparison of the state-of-the-art methods in neural network verification. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this competition.
• Score: 1.4824891788575418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This report summarizes the second International Verification of Neural Networks Competition (VNN-COMP 2021), held as a part of the 4th Workshop on Formal Methods for ML-Enabled Autonomous Systems that was collocated with the 33rd International Conference on Computer-Aided Verification (CAV). Twelve teams participated in this competition. The goal of the competition is to provide an objective comparison of the state-of-the-art methods in neural network verification, in terms of scalability and speed. Along this line, we used standard formats (ONNX for neural networks and VNNLIB for specifications), standard hardware (all tools are run by the organizers on AWS), and tool parameters provided by the tool authors. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this competition.

Related papers

• Competitive Programming with Large Reasoning Models [73.7455809592467]
  We show that reinforcement learning applied to large language models (LLMs) significantly boosts performance on complex coding and reasoning tasks. We compare two general-purpose reasoning models - OpenAI o1 and an early checkpoint of o3 - with a domain-specific system, o1-ioi. Our findings show that although specialized pipelines such as o1-ioi yield solid improvements, the scaled-up, general-purpose o3 model surpasses those results without relying on hand-crafted inferences.
  arXiv  Detail & Related papers  (2025-02-03T23:00:15Z)
• The Fifth International Verification of Neural Networks Competition (VNN-COMP 2024): Summary and Results [3.9189620165765]
  This report summarizes the 5th International Verification of Neural Networks Competition (VNN-COMP 2024) VNN-COMP is held annually to facilitate the fair and objective comparison of state-of-the-art neural network verification tools. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this iteration of this competition.
  arXiv  Detail & Related papers  (2024-12-28T03:07:00Z)
• The Fourth International Verification of Neural Networks Competition (VNN-COMP 2023): Summary and Results [7.3262152011453745]
  This report summarizes the 4th International Verification of Neural Networks Competition (VNN-COMP 2023) VNN-COMP is held annually to facilitate the fair and objective comparison of state-of-the-art neural network verification tools. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this iteration of this competition.
  arXiv  Detail & Related papers  (2023-12-28T00:46:35Z)
• Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO [50.58083807719749]
  We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.
  arXiv  Detail & Related papers  (2023-08-30T07:16:11Z)
• First Three Years of the International Verification of Neural Networks Competition (VNN-COMP) [9.02791567988691]
  In the VNN-COMP, participants submit software tools that analyze whether given neural networks satisfy specifications describing their input-output behavior. We summarize the key processes, rules, and results, present trends observed over the last three years, and provide an outlook into possible future developments.
  arXiv  Detail & Related papers  (2023-01-14T04:04:12Z)
• The Third International Verification of Neural Networks Competition (VNN-COMP 2022): Summary and Results [9.02791567988691]
  This report summarizes the 3rd International Verification of Neural Networks Competition (VNN-COMP 2022) VNN-COMP is held annually to facilitate the fair and objective comparison of state-of-the-art neural network verification tools. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this iteration of this competition.
  arXiv  Detail & Related papers  (2022-12-20T15:58:01Z)
• Batch-Ensemble Stochastic Neural Networks for Out-of-Distribution Detection [55.028065567756066]
  Out-of-distribution (OOD) detection has recently received much attention from the machine learning community due to its importance in deploying machine learning models in real-world applications. In this paper we propose an uncertainty quantification approach by modelling the distribution of features. We incorporate an efficient ensemble mechanism, namely batch-ensemble, to construct the batch-ensemble neural networks (BE-SNNs) and overcome the feature collapse problem. We show that BE-SNNs yield superior performance on several OOD benchmarks, such as the Two-Moons dataset, the FashionMNIST vs MNIST dataset, FashionM
  arXiv  Detail & Related papers  (2022-06-26T16:00:22Z)
• Competing Mutual Information Constraints with Stochastic Competition-based Activations for Learning Diversified Representations [5.981521556433909]
  This work aims to address the long-established problem of learning diversified representations. We combine information-theoretic arguments with competition-based activations. As we experimentally show, the resulting networks yield significant discnative representation learning abilities.
  arXiv  Detail & Related papers  (2022-01-10T20:12:13Z)
• Analysing Affective Behavior in the second ABAW2 Competition [70.86998050535944]
  The Affective Behavior Analysis in-the-wild (ABAW2) 2021 Competition is the second -- following the first very successful ABAW Competition held in conjunction with IEEE FG 2020- Competition that aims at automatically analyzing affect.
  arXiv  Detail & Related papers  (2021-06-14T11:30:19Z)
• A Two-Stage Approach to Device-Robust Acoustic Scene Classification [63.98724740606457]
  Two-stage system based on fully convolutional neural networks (CNNs) is proposed to improve device robustness. Our results show that the proposed ASC system attains a state-of-the-art accuracy on the development set. Neural saliency analysis with class activation mapping gives new insights on the patterns learnt by our models.
  arXiv  Detail & Related papers  (2020-11-03T03:27:18Z)
• When Does Self-Supervision Help Graph Convolutional Networks? [118.37805042816784]
  Self-supervision as an emerging technique has been employed to train convolutional neural networks (CNNs) for more transferrable, generalizable, and robust representation learning of images. In this study, we report the first systematic exploration of incorporating self-supervision into graph convolutional networks (GCNs) Our results show that, with properly designed task forms and incorporation mechanisms, self-supervision benefits GCNs in gaining more generalizability and robustness.
  arXiv  Detail & Related papers  (2020-06-16T13:29:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.