Recent years have seen a surge in the attention drawn by deep learning as its presence gets stronger in numerous application domains, including safety-critical systems such as autonomous vehicles. Although neural networks have superb generalization ability in object recognition, they are surprisingly susceptible to adversarial perturbations that are often imperceptible to the human eye and yet can trick the neural networks into a misclassifiaction. One promising approach to improving the robustness of neural networks is adversarial training, whereby neural networks are trained using adversarially perturbed examples as well as the original training data.
In evaluating the robustness of a neural network, it is a common practice to measure the zero-one loss of the neural network with respect to adversarially perturbed examples. However, because the zero-one loss function is not smooth, other loss functions are instead used to train neural networks. This raises the question of what loss function should be used in the training in order to provide the highest degree of robustness to neural networks. Focusing on a concrete variant of adversarial training called distributionally robust optimization (DRO), this work attempts to empirically answer this question.
The present work has found out that optimal loss functions for DRO vary from DRO algorithm to DRO algorithm. In addition, the cross entropy loss function, which is widely used in training neural networks, yields acceptable robustness in all three DRO algorithms considered in this work. Therefore, as far as the robustness of neural networks in DRO is concerned, there is no need to reconsider the standard use of the cross entropy loss function in training neural networks.