GLEAN: Generalized Deduplication Enabled Approximate Edge Analytics

TY - JOUR

T1 - GLEAN

T2 - Generalized Deduplication Enabled Approximate Edge Analytics

AU - Hurst, Aaron

AU - Lucani Rötter, Daniel Enrique

AU - Assent, Ira

AU - Zhang, Qi

PY - 2023/3/1

Y1 - 2023/3/1

N2 - The Internet of Things (IoT) has brought about exponential growth in sensor data. This has led to increasing demands for efficient and novel data transmission, storage, and analytics solutions for sustainable IoT ecosystems. It has been shown that the generalized deduplication (GD) compression algorithm offers not only competitive compression ratio and throughput but also random access properties that enable direct analytics of compressed data. In this article, we thoroughly stress test existing methods for direct analytics of GD compressed data with a diverse collection of 103 data sets, identify the need to optimize GD for analytics, and develop a new version of GD to this end. We also propose the generalized deduplication-enabled approximate edge analytics (GLEAN) framework. This framework applies the aforementioned analytics techniques at the Edge server to deliver end-to-end lossless data compression and high-quality Edge analytics in the IoT, thereby addressing challenges related to data transmission, storage, and analytics. Impressive analytics performance was achieved using this framework, with a median increase in k -means clustering error of just 2% relative to analytics performed on uncompressed data, while running 7.5× faster and requiring 3.9× less storage at the Edge server compared to universal compressors.

AB - The Internet of Things (IoT) has brought about exponential growth in sensor data. This has led to increasing demands for efficient and novel data transmission, storage, and analytics solutions for sustainable IoT ecosystems. It has been shown that the generalized deduplication (GD) compression algorithm offers not only competitive compression ratio and throughput but also random access properties that enable direct analytics of compressed data. In this article, we thoroughly stress test existing methods for direct analytics of GD compressed data with a diverse collection of 103 data sets, identify the need to optimize GD for analytics, and develop a new version of GD to this end. We also propose the generalized deduplication-enabled approximate edge analytics (GLEAN) framework. This framework applies the aforementioned analytics techniques at the Edge server to deliver end-to-end lossless data compression and high-quality Edge analytics in the IoT, thereby addressing challenges related to data transmission, storage, and analytics. Impressive analytics performance was achieved using this framework, with a median increase in k -means clustering error of just 2% relative to analytics performed on uncompressed data, while running 7.5× faster and requiring 3.9× less storage at the Edge server compared to universal compressors.

KW - Approximate analytics

KW - Internet of Things (IoT)

KW - data compression

KW - edge computing

UR - http://www.scopus.com/inward/record.url?scp=85128265367&partnerID=8YFLogxK

U2 - 10.1109/JIOT.2022.3166455

DO - 10.1109/JIOT.2022.3166455

M3 - Journal article

SN - 2327-4662

VL - 10

SP - 4006

EP - 4020

JO - IEEE Internet of Things Journal

JF - IEEE Internet of Things Journal

IS - 5

ER -