Super-convergence and Differential Privacy: Training faster with better privacy guarantees

TY - GEN

T1 - Super-convergence and Differential Privacy: Training faster with better privacy guarantees

AU - Frisk, Osvald

AU - Dormann, Friedrich

AU - Lillelund, Christian Marius

AU - Pedersen, Christian Fischer

PY - 2021/3/24

Y1 - 2021/3/24

N2 - The combination of deep neural networks and Differential Privacy has been of increasing interest in recent years, as it offers important data protection guarantees to the individuals of the training datasets used. However, using Differential Privacy in the training of neural networks comes with a set of shortcomings, like a decrease in validation accuracy and a significant increase in the use of resources and time in training. In this paper, we examine super-convergence as a way of greatly increasing training speed of differentially private neural networks, addressing the shortcoming of high training time and resource use. Super-convergence allows for acceleration in network training using very high learning rates, and has been shown to achieve models with high utility in orders of magnitude less training iterations than conventional ways. Experiments in this paper show that this order-of-magnitude speedup can also be seen when combining it with Differential Privacy, allowing for higher validation accuracies in much fewer training iterations compared to non-private, non-super convergent baseline models. Furthermore, super-convergence is shown to improve the privacy guarantees of private models.

AB - The combination of deep neural networks and Differential Privacy has been of increasing interest in recent years, as it offers important data protection guarantees to the individuals of the training datasets used. However, using Differential Privacy in the training of neural networks comes with a set of shortcomings, like a decrease in validation accuracy and a significant increase in the use of resources and time in training. In this paper, we examine super-convergence as a way of greatly increasing training speed of differentially private neural networks, addressing the shortcoming of high training time and resource use. Super-convergence allows for acceleration in network training using very high learning rates, and has been shown to achieve models with high utility in orders of magnitude less training iterations than conventional ways. Experiments in this paper show that this order-of-magnitude speedup can also be seen when combining it with Differential Privacy, allowing for higher validation accuracies in much fewer training iterations compared to non-private, non-super convergent baseline models. Furthermore, super-convergence is shown to improve the privacy guarantees of private models.

KW - Machine Learning, Differential Privacy, Rényi Differential Privacy, Deep Learning, Super-convergence

UR - http://www.scopus.com/inward/record.url?scp=85104989291&partnerID=8YFLogxK

M3 - Article in proceedings

SN - 978-1-6654-4844-4

SP - 1

EP - 6

BT - 55th Annual Conference on Information Sciences and Systems

PB - IEEE

T2 - 55th Annual Conference on Information Sciences and Systems

Y2 - 24 March 2021 through 26 March 2021

ER -