Direct Analytics of Generalized Deduplication Compressed IoT Data

TY - GEN

T1 - Direct Analytics of Generalized Deduplication Compressed IoT Data

AU - Hurst, Aaron

AU - Zhang, Qi

AU - Lucani Rötter, Daniel Enrique

AU - Assent, Ira

N1 - Conference code: 2021

PY - 2021

Y1 - 2021

N2 - Given the ever increasing volume of data generated by the Internet of Things, data compression plays an essential role in reducing the cost of data transmission and storage. However, it also introduces a barrier, namely decompression, between users and the data-driven insights they require. We propose methods for direct analytics of compressed data based on the Generalized Deduplication compression algorithm. When applied to data clustering, the accuracy of the best performing method differs by merely 1-5% when compared to analytics performed upon the uncompressed data. However, it runs four times faster, accesses only 14% as much data and requires significantly less storage since the data is always compressed. These results show that it is possible to simultaneously reap the benefits of compression and accurate, high-speed analytics in many applications.

AB - Given the ever increasing volume of data generated by the Internet of Things, data compression plays an essential role in reducing the cost of data transmission and storage. However, it also introduces a barrier, namely decompression, between users and the data-driven insights they require. We propose methods for direct analytics of compressed data based on the Generalized Deduplication compression algorithm. When applied to data clustering, the accuracy of the best performing method differs by merely 1-5% when compared to analytics performed upon the uncompressed data. However, it runs four times faster, accesses only 14% as much data and requires significantly less storage since the data is always compressed. These results show that it is possible to simultaneously reap the benefits of compression and accurate, high-speed analytics in many applications.

KW - Internet of Things

KW - clustering methods

KW - data compression

KW - data mining

KW - explainable AI

UR - http://www.scopus.com/inward/record.url?scp=85184369178&partnerID=8YFLogxK

U2 - 10.1109/GLOBECOM46510.2021.9685589

DO - 10.1109/GLOBECOM46510.2021.9685589

M3 - Article in proceedings

T3 - IEEE Global Communications Conference (GLOBECOM)

BT - 2021 IEEE Global Communications Conference, GLOBECOM 2021 - Proceedings

PB - IEEE

T2 - IEEE Conference and Exhibition on Global Telecommunications

Y2 - 7 December 2021 through 11 December 2021

ER -