Cross-File Deduplication (FastCDC)
What Is Content-Defined Deduplication?
Traditional compression algorithms (like LZMA, DEFLATE, and BZIP2) operate on individual files or solid stream blocks. If you archive 10 versions of a document, a folder containing duplicated libraries, or daily virtual machine disk snapshots, traditional tools compress the redundant data over and over again.
DodaZIP incorporates FastCDC (Fast Content-Defined Chunking) with cryptographic BLAKE3 hashing to achieve cross-file block deduplication. Identical data blocks—even if located inside completely different files or shifted by a few bytes—are stored only once.
⚡ How FastCDC Works
Traditional fixed-block deduplication splits files at fixed byte intervals (e.g. every 64 KB). If even a single byte is inserted at the beginning of a file, every subsequent block boundary shifts, causing 100% deduplication failure.
FastCDC avoids this problem using rolling hashes:
- A rolling hash scans the data byte by byte.
- When the hash matches specific mathematical bit-patterns, a chunk boundary is declared.
- Because boundaries are determined by the content rather than fixed file offsets, inserting or deleting bytes only alters the immediate chunk—all surrounding chunks retain identical boundaries and hashes.
File A: [ Chunk 1 ] [ Chunk 2 ] [ Chunk 3 ]
File B: [ Inserted Byte ] [ Chunk 1 ] [ Chunk 2 ] [ Chunk 3 ]
Result: Chunk 1, Chunk 2, and Chunk 3 are recognized as identical and stored ONCE!🧬 Content-Addressed Chunk Table
When creating a .dzip archive, DodaZIP builds an internal index table:
- BLAKE3 Hash: Unique 256-bit cryptographic fingerprint per chunk.
- Reference Counting: Tracks how many files and locations utilize the same chunk.
- Physical Offset: The exact compressed block location in the archive payload.
During extraction, DodaZIP reassembles files by streaming their corresponding chunk references in parallel.
💰 Real-World Space & Cost Savings
| Workload | Traditional ZIP/7z Size | DodaZip FastCDC Size | Space Saved |
|---|---|---|---|
| Daily SQL Database Dumps (7 days) | ~35 GB | ~6.2 GB | 82% |
| Virtual Machine Snapshots (.vmdk / .qcow2) | ~120 GB | ~28 GB | 76% |
Git Repositories with node_modules | ~4.5 GB | ~1.1 GB | 75% |
| Iterative Document / Backup Versions | ~18 GB | ~4.8 GB | 73% |
💻 CLI Usage
# Create an archive with FastCDC deduplication enabled (default for .dzip)
dodazip c -i ./daily-backups -o backups.dzip --codec zstd
# Check deduplication ratio and savings summary
dodazip l backups.dzip --verbose