A Selective Training Approach for Very Fast Backpropagation on Sentence Embeddings

Research output: Contribution to book/anthology/report/proceedingArticle in proceedingsResearchpeer-review

Distributed word embeddings and related models have been shown to successfully learn natural language representations. However, for complex models on large datasets training time can be extensive, approaching weeks, which is often infeasible in practice.

In this work, we present a novel method to reduce training time substantially by selecting training instances that provide relevant information for training. Selection is based on the similarity of the learned representations over input instances, thus allowing for learning a non-trivial weighting scheme from multi-dimensional representations.

We demonstrate the efficiency and effectivity of our approach in several text classification tasks using recursive neural networks. Our empirical evaluation shows that the objective function converges up to 6 times faster without sacrificing accuracy.
Original languageEnglish
Title of host publicationProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing : EMNLP '18
Number of pages9
Publication year23 Jan 2019
Publication statusSubmitted - 23 Jan 2019

See relations at Aarhus University Citationformats

ID: 130411244