Generalized Deduplication: Lossless Compression for Large Amounts of Small IoT Data

TY - GEN

T1 - Generalized Deduplication

T2 - European Wireless 2019 - 25th European Wireless Conference

AU - Vestergaard, Rasmus

AU - Lucani Rötter, Daniel Enrique

AU - Zhang, Qi

PY - 2019

Y1 - 2019

N2 - We show that a generalization of deduplication can enable compressed storage of sensor data. The method uses error- correcting codes in a non-traditional manner to identify similar elements, and then leverages this similarity for compression. Using Reed Solomon codes, our method has a theoretical potential to reduce the cost of storing chunks of 16 bytes to as much as 5 times less, and up to 65 times less for chunks of 255 bytes. We define a simple model for sensor data, and show how our approach is able to compress data from the model, realizing its compression potential with much smaller data sets than classic deduplication requires. This demonstrates that generalized deduplication can be a viable solution for practical lossless compression of small IoT data in scenarios where classic deduplication is ineffective.

AB - We show that a generalization of deduplication can enable compressed storage of sensor data. The method uses error- correcting codes in a non-traditional manner to identify similar elements, and then leverages this similarity for compression. Using Reed Solomon codes, our method has a theoretical potential to reduce the cost of storing chunks of 16 bytes to as much as 5 times less, and up to 65 times less for chunks of 255 bytes. We define a simple model for sensor data, and show how our approach is able to compress data from the model, realizing its compression potential with much smaller data sets than classic deduplication requires. This demonstrates that generalized deduplication can be a viable solution for practical lossless compression of small IoT data in scenarios where classic deduplication is ineffective.

M3 - Article in proceedings

SN - 978-3-8007-4948-5

SP - 67

EP - 71

BT - European Wireless Conference

PB - VDE Verlag GmbH

Y2 - 2 May 2019 through 5 May 2019

ER -