Metrics and methods for robustness evaluation of neural networks with generative models

Igor Buzhinsky*, Arseny Nerinovsky, Stavros Tripakis

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

3 Citations (Scopus)


Recent studies have shown that modern deep neural network classifiers are easy to fool, assuming that an adversary is able to slightly modify their inputs. Many papers have proposed adversarial attacks, defenses and methods to measure robustness to such adversarial perturbations. However, most commonly considered adversarial examples are based on perturbations in the input space of the neural network that are unlikely to arise naturally. Recently, especially in computer vision, researchers discovered “natural” perturbations, such as rotations, changes of brightness, or more high-level changes, but these perturbations have not yet been systematically used to measure the performance of classifiers. In this paper, we propose several metrics to measure robustness of classifiers to natural adversarial examples, and methods to evaluate them. These metrics, called latent space performance metrics, are based on the ability of generative models to capture probability distributions. On four image classification case studies, we evaluate the proposed metrics for several classifiers, including ones trained in conventional and robust ways. We find that the latent counterparts of adversarial robustness are associated with the accuracy of the classifier rather than its conventional adversarial robustness, but the latter is still reflected on the properties of found latent perturbations. In addition, our novel method of finding latent adversarial perturbations demonstrates that these perturbations are often perceptually small.

Original languageEnglish
Pages (from-to)3977-4012
Number of pages36
JournalMachine Learning
Issue number10
Early online date6 Jul 2021
Publication statusPublished - Oct 2023
MoE publication typeA1 Journal article-refereed


  • Adversarial examples
  • Generative models
  • Natural adversarial examples
  • Reliable machine learning


Dive into the research topics of 'Metrics and methods for robustness evaluation of neural networks with generative models'. Together they form a unique fingerprint.

Cite this