DeepTest: automated testing of deep-neural-network-driven autonomous cars
- Year
- 2018
- Authors
- Yuchi Tian, Kexin Pei, Suman Jana, Baishakhi Ray
- DOI
- 10.1145/3180155.3180220
Related
Persistent Notes
In-text annotations
“In this paper, we design, implement, and evaluate DeepTest, a systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles that can potentially lead to fatal crashes." Page 303
“1 INTRODUCTION” Page 303
“Moreover, the Satisfiability Modulo Theory (SMT) solvers that have been quite successful at generating high-coverage test inputs for traditional software are known to have trouble with formulas involving floating-point arithmetic and highly nonlinear constraints, which are commonly used in DNNs." Page 304
“irst, we leverage the notion of neuron coverage (i.e., the number of neurons activated by a set of test inputs) to systematically explore different parts of the DNN logic." Page 304
“2 BACKGROUND” Page 304
“2.1 Deep Learning for Autonomous Driving” Page 304
“2.2 Different DNN Architectures” Page 305
“3 METHODOLOGY” Page 306
“3.1 Systematic Testing with Neuron Coverage” Page 306
“It is defined as the ratio of unique neurons that get activated for given input(s) and the total number of neurons in a DNN” Page 306
“An individual neuron is considered activated if the neuron’s output (scaled by the overall layer’s outputs) is larger than a DNN-wide threshold. In this paper, we use 0.2 as the neuron activation threshold for all our experiments." Page 306
“3.2 Increasing Coverage with Synthetic Images” Page 306
“Therefore, DeepTest focuses on generating realistic synthetic images by applying image transformations on seed images and mimic different real-world phenomena like camera lens distortions, object movements, different weather conditions, etc. To this end, we investigate nine different realistic image transformations (changing brightness, changing contrast, translation, scaling, horizontal shearing, rotation, blurring, fog effect, and rain effect)." Page 306
“inear, affine, and convolutional." Page 306
“3.3 Combining Transformations to Increase Coverage” Page 307
“As the individual image transformations increase neuron coverage, one obvious question is whether they can be combined to further increase the neuron coverage." Page 307
“3.4 Creating a Test Oracle with Metamorphic Relations” Page 307
“we leverage metamorphic relations [33] between the car behaviors across different synthetic images." Page 307
“For example, the autonomous car’s steering angle should not change significantly for the same image under any lighting/weather conditions, blurring, or any affine transformations with small parameter values." Page 307
“The above equation assumes that the errors produced by a model for the transformed images as input should be within a range of λ times the MSE produced by the original image set." Page 307
“4 IMPLEMENTATION” Page 307
“5 RESULTS” Page 308
“As steering angle is a continuous variable, we check Spearman rank correlation [76] between neuron coverage and steering angle” Page 308
“We use the Wilcoxon nonparametric test as the steering direction can only have two values (left and right)." Page 308
“Neuron coverage is correlated with input-output diversity and can be used to systematic test generation." Page 309
“Different image transformations tend to activate different sets of neurons." Page 309
“By systematically combining different image transformations, neuron coverage can be improved by around 100% w.r.t. the coverage achieved by the original seed images." Page 310
“Accuracy of a DNN can be improved up to 46% by retraining the DNN with synthetic data generated by DeepTest." Page 311
“6 THREATS TO VALIDITY” Page 312
“We restricted ourselves to only test the accuracy of the steering angle as our tested models do not support braking and acceleration yet." Page 312
“7 RELATED WORK” Page 312
“8 CONCLUSION” Page 312
“9 ACKNOWLEDGEMENTS” Page 312
%% Import Date: 2026-02-02T11:34:38.193-05:00 %%