Architecture Selection via the Trade-off Between Accuracy and Robustness

• URL: http://arxiv.org/abs/1906.01354v2
• Date: Fri, 23 May 2025 12:04:35 GMT
• Title: Architecture Selection via the Trade-off Between Accuracy and Robustness
• Authors: Zhun Deng, Cynthia Dwork, Jialiang Wang, Yao Zhao,
• Abstract summary: We propose a method and define quantities to characterize the trade-off between accuracy and robustness for a given architecture.<n>Specifically we introduce a simple trade-off curve, define and study an influence function that captures the sensitivity, under adversarial attack, of the optima of a given loss function.<n>In experiments, we demonstrate the corresponding trade-off curves of neural networks and how they vary with respect to factors such as number of layers, neurons, and across different network structures.
• Score: 48.39317297503043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We provide a general framework for characterizing the trade-off between accuracy and robustness in supervised learning. We propose a method and define quantities to characterize the trade-off between accuracy and robustness for a given architecture, and provide theoretical insight into the trade-off. Specifically we introduce a simple trade-off curve, define and study an influence function that captures the sensitivity, under adversarial attack, of the optima of a given loss function. We further show how adversarial training regularizes the parameters in an over-parameterized linear model, recovering the LASSO and ridge regression as special cases, which also allows us to theoretically analyze the behavior of the trade-off curve. In experiments, we demonstrate the corresponding trade-off curves of neural networks and how they vary with respect to factors such as number of layers, neurons, and across different network structures. Such information provides a useful guideline to architecture selection.

Related papers

• Sensitivity of Stability: Theoretical & Empirical Analysis of Replicability for Adaptive Data Selection in Transfer Learning [2.5526642520700946]
  We introduce a mathematical framework that quantifies the fundamental trade-off between adaptation effectiveness and result consistency.<n>We show that highly adaptive strategies like gradient-based and curriculum learning achieve superior task performance but suffer from high replicability failure rates.<n>We also show that source domain pretraining provides a powerful mitigation mechanism, reducing failure rates by up to 30% while preserving performance gains.
  arXiv  Detail & Related papers  (2025-08-06T21:56:56Z)
• Neural Networks Decoded: Targeted and Robust Analysis of Neural Network Decisions via Causal Explanations and Reasoning [9.947555560412397]
  We introduce TRACER, a novel method grounded in causal inference theory to estimate the causal dynamics underpinning DNN decisions. Our approach systematically intervenes on input features to observe how specific changes propagate through the network, affecting internal activations and final outputs. TRACER further enhances explainability by generating counterfactuals that reveal possible model biases and offer contrastive explanations for misclassifications.
  arXiv  Detail & Related papers  (2024-10-07T20:44:53Z)
• Large Margin Discriminative Loss for Classification [3.3975558777609915]
  We introduce a novel discriminative loss function with large margin in the context of Deep Learning.<n>This loss boosts the discriminative power of neural networks represented by intra-class compactness and inter-class separability.<n>We design a strategy called partial momentum updating that enjoys simultaneously stability and consistency in training.
  arXiv  Detail & Related papers  (2024-05-28T18:10:45Z)
• Understanding Robust Overfitting from the Feature Generalization Perspective [61.770805867606796]
  Adversarial training (AT) constructs robust neural networks by incorporating adversarial perturbations into natural data. It is plagued by the issue of robust overfitting (RO), which severely damages the model's robustness. In this paper, we investigate RO from a novel feature generalization perspective.
  arXiv  Detail & Related papers  (2023-10-01T07:57:03Z)
• Mutual Information Estimation via $f$-Divergence and Data Derangements [6.43826005042477]
  We propose a novel class of discrimi mutual information estimators based on the variational representation of the $f$-divergence. The proposed estimator is flexible since it exhibits an excellent bias/ variance trade-off.
  arXiv  Detail & Related papers  (2023-05-31T16:54:25Z)
• Fundamental Limits and Tradeoffs in Invariant Representation Learning [99.2368462915979]
  Many machine learning applications involve learning representations that achieve two competing goals. Minimax game-theoretic formulation represents a fundamental tradeoff between accuracy and invariance. We provide an information-theoretic analysis of this general and important problem under both classification and regression settings.
  arXiv  Detail & Related papers  (2020-12-19T15:24:04Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)
• Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness [97.67477497115163]
  We use mode connectivity to study the adversarial robustness of deep neural networks. Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.
  arXiv  Detail & Related papers  (2020-04-30T19:12:50Z)
• Precise Tradeoffs in Adversarial Training for Linear Regression [55.764306209771405]
  We provide a precise and comprehensive understanding of the role of adversarial training in the context of linear regression with Gaussian features. We precisely characterize the standard/robust accuracy and the corresponding tradeoff achieved by a contemporary mini-max adversarial training approach. Our theory for adversarial training algorithms also facilitates the rigorous study of how a variety of factors (size and quality of training data, model overparametrization etc.) affect the tradeoff between these two competing accuracies.
  arXiv  Detail & Related papers  (2020-02-24T19:01:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.