Aarhus University Seal

Continual Transformers: Redundancy-Free Attention for Online Inference

Research output: Working paper/Preprint Working paperResearchpeer-review

Transformers in their common form are inherently limited to operate on whole token sequences rather than on one token at a time. Consequently, their use during online inference on time-series data entails considerable redundancy due to the overlap in successive token sequences. In this work, we propose novel formulations of the Scaled Dot-Product Attention, which enable Transformers to perform efficient online token-by-token inference on a continual input stream. Importantly, our modifications are purely to the order of computations, while the outputs and learned weights are identical to those of the original Transformer Encoder. We validate our Continual Transformer Encoder with experiments on the THUMOS14, TVSeries and GTZAN datasets with remarkable results: Our Continual one- and two-block architectures reduce the floating point operations per prediction by up to 63x and 2.6x, respectively, while retaining predictive performance.
Original languageEnglish
PublisherThe Eleventh International Conference on Learning Representations
Number of pages15
Publication statusPublished - Nov 2022

See relations at Aarhus University Citationformats

ID: 303920882