Deduplication of Textual Data by NLP Approaches

TY - GEN

T1 - Deduplication of Textual Data by NLP Approaches

AU - Ghassabi, Kiana

AU - Pahlevani, Peyman

AU - Lucani Rötter, Daniel Enrique

PY - 2023/8

Y1 - 2023/8

N2 - With the increasing amount of digital data, data deduplication has become an increasingly popular method for reducing data in large-scale storage systems. Generalized deduplication is an alternative technique for reducing the cost of data storage by identifying similar data chunks. This paper proposes TL-GD, a method for improving cloud storage efficiency using generalized deduplication focusing on textual datasets. The core concept of this study is to develop an efficient deduplication system that combines an alternative technique for splitting data into smaller pieces and a new approach for transforming data pieces into bases and deviations. The performance of the system has been validated using two real-world datasets. We also compare the results to state-of-the-art deduplication methods. Our evaluation results show that TL-GD achieves nearly 67% lossless compression for textual navigation instructions datasets, which is a 25% improvement on average compared to existing deduplication techniques.

AB - With the increasing amount of digital data, data deduplication has become an increasingly popular method for reducing data in large-scale storage systems. Generalized deduplication is an alternative technique for reducing the cost of data storage by identifying similar data chunks. This paper proposes TL-GD, a method for improving cloud storage efficiency using generalized deduplication focusing on textual datasets. The core concept of this study is to develop an efficient deduplication system that combines an alternative technique for splitting data into smaller pieces and a new approach for transforming data pieces into bases and deviations. The performance of the system has been validated using two real-world datasets. We also compare the results to state-of-the-art deduplication methods. Our evaluation results show that TL-GD achieves nearly 67% lossless compression for textual navigation instructions datasets, which is a 25% improvement on average compared to existing deduplication techniques.

KW - CSP

KW - Generalized Deduplication

KW - Storage

U2 - 10.1109/VTC2023-Spring57618.2023.10199538

DO - 10.1109/VTC2023-Spring57618.2023.10199538

M3 - Article in proceedings

T3 - I E E E V T S Vehicular Technology Conference. Proceedings

BT - 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring)

PB - IEEE

ER -