SGD Biased towards Early Important Samples for Efficient Training

TY - GEN

T1 - SGD Biased towards Early Important Samples for Efficient Training

AU - Quercia, Alessio

AU - Morrison, Abigail

AU - Scharr, Hanno

AU - Assent, Ira

PY - 2023

Y1 - 2023

N2 - In deep learning, using larger training datasets usually leads to more accurate models. However, simply adding more but redundant data may be inefficient, as some training samples may be more informative than others. We propose to bias SGD (Stochastic Gradient Descent) towards samples that are found to be more important after a few training epochs, by sampling them more often for the rest of training.In contrast to state-of-the-art, our approach requires less computational overhead to estimate sample importance, as it computes estimates once during training using the prediction probabilities, and does not require that training be restarted.In the experimental evaluation, we see that our learning technique trains faster than state-of-the-art and can achieve higher test accuracy, especially when datasets are not well balanced. Lastly, results suggest that our approach has intrinsic balancing properties. Code is available at https://github.com/AlessioQuercia/sgd_biased.

AB - In deep learning, using larger training datasets usually leads to more accurate models. However, simply adding more but redundant data may be inefficient, as some training samples may be more informative than others. We propose to bias SGD (Stochastic Gradient Descent) towards samples that are found to be more important after a few training epochs, by sampling them more often for the rest of training.In contrast to state-of-the-art, our approach requires less computational overhead to estimate sample importance, as it computes estimates once during training using the prediction probabilities, and does not require that training be restarted.In the experimental evaluation, we see that our learning technique trains faster than state-of-the-art and can achieve higher test accuracy, especially when datasets are not well balanced. Lastly, results suggest that our approach has intrinsic balancing properties. Code is available at https://github.com/AlessioQuercia/sgd_biased.

KW - Deep Learning

KW - Hard Example Mining

KW - Optimization

UR - https://www.scopus.com/pages/publications/85185392437

U2 - 10.1109/ICDM58522.2023.00163

DO - 10.1109/ICDM58522.2023.00163

M3 - Article in proceedings

SP - 1289

EP - 1294

BT - IEEE International Conference on Data Mining, ICDM 2023, Shanghai, China, December 1-4, 2023

A2 - Chen, Guihai

A2 - Khan, Latifur

A2 - Gao, Xiaofeng

A2 - Qiu, Meikang

A2 - Pedrycz, Witold

A2 - Wu, Xindong

PB - IEEE

ER -