Deduplication exploits the presence of similar data chunks to reduce storage overhead. Generalised deduplication (GD) uses transformation functions to split data into a basis (common to millions of chunks) and a deviation with respect to the basis. Doing so, it avoids computing additional hashes, comparing or differentiating against previously stored chunks. MINERVAFS is the first FUSE-based file system for GD. We implement and evaluate it using several real-world datasets, e.g., satellite images and virtual machine images, comparing against classical deduplication approaches (ZFS, SDFS), delta compression (xdelta) or compression (Gzip). Compared to ZFS, MINERVAFS achieves up to 63.53% (average of 27.38%) saving in storage usage and a speedup of 16% in read-heavy workloads. For VM images, MINERVAFS’s data compression is on par with Gzip, while outperforming ZFS by severalfold. In contrast to ZFS’ growing RAM costs when more data is stored, MinervaFS’ RAM usage is independent from the amount of data stored, making it well suited to handle growing storage demands.
Original language
English
Title of host publication
International Symposium on Reliable Distributed Systems (SRDS)
