Canonical Digests for Compressed Archives

Abstract

Two archives that compress the same original files will have different on-disk byte representations if they are created with different archive formats (e.g., ZIP, TAR) or compression algorithms (e.g., LZMA, DEFLATE). The ability to summarize the content and metadata of a compressed archive for later content or metadata identity analyses can help systems operate more efficiently; scan-repack-and-forward caching internet middleboxes, information processing systems, and file management and deduplication systems can all benefit from the offline ability to identify similar archives despite differences in on-disk byte representations. Many compressed archive formats are in use today, each with its own binary format, metadata fields, and often inconsistent implementations, so comparison and summarization must be flexible and format agnostic. To enable similarity comparisons between compressed archives, we propose a format-agnostic tiered canonical digest with 8 content and metadata parts. The digest's output size is independent of archive format, compression algorithm, and archive size. The digests reveal identity and similarity between archives and gracefully supports future extensions. They also enable new features such as flexible integrity checks, smart file caching, remote file system attestation, and forensic similarity analysis. We implement a reference canonical digest tool that supports 13 modern and historical compressed archive formats for digest creation, validation, and similarity analysis. Performance measurements show that digests are typically 350-440 bytes, can be computed in less than 1.5ms per KB of compressed archive, and use 300-500 KB of memory per KB of compressed archive.