Model Counting in the Wild

• URL: http://arxiv.org/abs/2408.07059v1
• Date: Tue, 13 Aug 2024 17:49:46 GMT
• Title: Model Counting in the Wild
• Authors: Arijit Shaw, Kuldeep S. Meel,
• Abstract summary: We conduct a rigorous assessment of the scalability of model counters in the wild. We evaluate six state-of-the-art model counters on these instances to assess scalability and runtime performance. Our analysis highlights the challenges and opportunities for portfolio-based approaches in model counting.
• Score: 31.05707402954459
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model counting is a fundamental problem in automated reasoning with applications in probabilistic inference, network reliability, neural network verification, and more. Although model counting is computationally intractable from a theoretical perspective due to its #P-completeness, the past decade has seen significant progress in developing state-of-the-art model counters to address scalability challenges. In this work, we conduct a rigorous assessment of the scalability of model counters in the wild. To this end, we surveyed 11 application domains and collected an aggregate of 2262 benchmarks from these domains. We then evaluated six state-of-the-art model counters on these instances to assess scalability and runtime performance. Our empirical evaluation demonstrates that the performance of model counters varies significantly across different application domains, underscoring the need for careful selection by the end user. Additionally, we investigated the behavior of different counters with respect to two parameters suggested by the model counting community, finding only a weak correlation. Our analysis highlights the challenges and opportunities for portfolio-based approaches in model counting.

Related papers

• Predictive Churn with the Set of Good Models [64.05949860750235]
  We study the effect of conflicting predictions over the set of near-optimal machine learning models. We present theoretical results on the expected churn between models within the Rashomon set. We show how our approach can be used to better anticipate, reduce, and avoid churn in consumer-facing applications.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• GEO-Bench: Toward Foundation Models for Earth Monitoring [139.77907168809085]
  We propose a benchmark comprised of six classification and six segmentation tasks. This benchmark will be a driver of progress across a variety of Earth monitoring tasks.
  arXiv  Detail & Related papers  (2023-06-06T16:16:05Z)
• Rigorous Assessment of Model Inference Accuracy using Language Cardinality [5.584832154027001]
  We develop a systematic approach that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools.
  arXiv  Detail & Related papers  (2022-11-29T21:03:26Z)
• Are Neural Topic Models Broken? [81.15470302729638]
  We study the relationship between automated and human evaluation of topic models. We find that neural topic models fare worse in both respects compared to an established classical method.
  arXiv  Detail & Related papers  (2022-10-28T14:38:50Z)
• Exploring the Trade-off between Plausibility, Change Intensity and Adversarial Power in Counterfactual Explanations using Multi-objective Optimization [73.89239820192894]
  We argue that automated counterfactual generation should regard several aspects of the produced adversarial instances. We present a novel framework for the generation of counterfactual examples.
  arXiv  Detail & Related papers  (2022-05-20T15:02:53Z)
• Consistent Counterfactuals for Deep Models [25.1271020453651]
  Counterfactual examples are used to explain predictions of machine learning models in key areas such as finance and medical diagnosis. This paper studies the consistency of model prediction on counterfactual examples in deep networks under small changes to initial training conditions.
  arXiv  Detail & Related papers  (2021-10-06T23:48:55Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)
• Amortized Bayesian Inference for Models of Cognition [0.1529342790344802]
  Recent advances in simulation-based inference using specialized neural network architectures circumvent many previous problems of approximate Bayesian computation. We provide a general introduction to amortized Bayesian parameter estimation and model comparison.
  arXiv  Detail & Related papers  (2020-05-08T08:12:15Z)
• Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples [84.8370546614042]
  Black-box nature of Deep Learning models has posed unanswered questions about what they learn from data. Generative Adversarial Network (GAN) and multi-objectives are used to furnish a plausible attack to the audited model. Its utility is showcased within a human face classification task, unveiling the enormous potential of the proposed framework.
  arXiv  Detail & Related papers  (2020-03-25T11:08:56Z)
• Context-dependent self-exciting point processes: models, methods, and risk bounds in high dimensions [21.760636228118607]
  High-dimensional autoregressive point processes model how current events trigger or inhibit future events, such as activity by one member of a social network can affect the future activity of his or her neighbors. We leverage ideas from compositional time series and regularization methods in machine learning to conduct network estimation for high-dimensional marked point processes.
  arXiv  Detail & Related papers  (2020-03-16T20:22:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.