Single-layer vision transformers for more accurate early exits with less overhead

Arian Bakhtiarnia; Qi Zhang; Alexandros Iosifidis

doi:10.1016/j.neunet.2022.06.038

Single-layer vision transformers for more accurate early exits with less overhead

Arian Bakhtiarnia^*
, Qi Zhang^*
, Alexandros Iosifidis^*

^*Corresponding author af dette arbejde

Institut for Elektro- og Computerteknologi - Communication, Control and Automation

Publikation: Bidrag til tidsskrift/Konferencebidrag i tidsskrift /Bidrag til avis › Tidsskriftartikel › Forskning › peer review

29 Citationer (Scopus)

118 Downloads (Pure)

Abstract

Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well as audiovisual crowd counting, we show that our method works for both classification and regression problems, and in both single- and multi-modal settings. Additionally, we introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis, that can lead to a more fine-grained dynamic inference.

Originalsprog	Engelsk
Tidsskrift	Neural Networks
Vol/bind	153
Sider (fra-til)	461-473
Antal sider	13
ISSN	0893-6080
DOI	https://doi.org/10.1016/j.neunet.2022.06.038
Status	Udgivet - sep. 2022

Adgang til dokumentet

10.1016/j.neunet.2022.06.038

1-s2.0-S0893608022002532-mainForlagets udgivne version, 1,36 MBLicens: CC BY-NC-ND

https://www.sciencedirect.com/science/article/pii/S0893608022002532?via%3Dihub

Biblioteksadgang?

Tjek tilgængelighed hos det Kgl. Bibliotek

Andre filer og links

Link to publication in Scopus

Citationsformater

@article{897075c1633546c5bbf9dd7e2183e228,

title = "Single-layer vision transformers for more accurate early exits with less overhead",

abstract = "Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well as audiovisual crowd counting, we show that our method works for both classification and regression problems, and in both single- and multi-modal settings. Additionally, we introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis, that can lead to a more fine-grained dynamic inference.",

keywords = "Dynamic inference, Early exiting, Multi-exit architecture, Multimodal deep learning, Vision transformer",

author = "Arian Bakhtiarnia and Qi Zhang and Alexandros Iosifidis",

year = "2022",

month = sep,

doi = "10.1016/j.neunet.2022.06.038",

language = "English",

volume = "153",

pages = "461--473",

journal = "Neural Networks",

issn = "0893-6080",

publisher = "Elsevier Ltd",

}

TY - JOUR

T1 - Single-layer vision transformers for more accurate early exits with less overhead

AU - Bakhtiarnia, Arian

AU - Zhang, Qi

AU - Iosifidis, Alexandros

PY - 2022/9

Y1 - 2022/9

N2 - Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well as audiovisual crowd counting, we show that our method works for both classification and regression problems, and in both single- and multi-modal settings. Additionally, we introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis, that can lead to a more fine-grained dynamic inference.

AB - Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well as audiovisual crowd counting, we show that our method works for both classification and regression problems, and in both single- and multi-modal settings. Additionally, we introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis, that can lead to a more fine-grained dynamic inference.

KW - Dynamic inference

KW - Early exiting

KW - Multi-exit architecture

KW - Multimodal deep learning

KW - Vision transformer

UR - https://www.scopus.com/pages/publications/85133931261

U2 - 10.1016/j.neunet.2022.06.038

DO - 10.1016/j.neunet.2022.06.038

M3 - Journal article

C2 - 35816859

SN - 0893-6080

VL - 153

SP - 461

EP - 473

JO - Neural Networks

JF - Neural Networks

ER -

Single-layer vision transformers for more accurate early exits with less overhead

Abstract

Adgang til dokumentet

Biblioteksadgang?

Andre filer og links

Fingeraftryk

Citationsformater