Testing Deep Neural Networks
summary
Empirical study evaluates DeepXplore, DLfuzz and DeepConcolic on MNIST/SVHN and finds they produce many invalid inputs — about 42% invalid on average, up to 73–100% in worst cases — which inflate neuron-coverage and other metrics. A solution embeds a variational autoencoder (and cross-checks with pixelCNN++ ) into test generation to constrain outputs to the training distribution; this eliminates most invalid tests, raises valid-test yield by >70%, and cuts average valid-test time from 6.2 to 1.5 minutes.
AI-written summary · about
chapters (18)
- 0:00Introduction and context
- 1:23Traditional software testing
- 3:40Deep neural network basics
- 6:55Challenges in DNN testing
- 9:12Valid vs invalid inputs
- 11:01Impact of invalid inputs
- 14:16Research objectives
- 17:18DNN coverage metrics
- 19:49Existing testing techniques
- 22:51Generating valid inputs
- 25:50VAE-based validation
- 28:35Experimental studies
- 30:45Test input validation results
- 34:49Effect of invalid inputs
- 38:05Proposed testing technique
- 42:22Technique evaluation
- 45:15Generative model validation
- 49:35Summary and future work
related talks
24mTesting Techniques for Critical Deep Learning Applications
59mStudying Failure Modes of Deep Learning and Ways To Fix Them
35mDeconstructing Deep Learning
20mIncremental Verification of Neural Networks
11mExploiting Verified Neural Networks via Floating Point Numerical Error
34mNeural networks: insane in the membrane
47mInterpreting Deep Neural Networks (DNNs)
31mWait, You Test Neural Networks Too?