Skip to content
Cross-File Deduplication (FastCDC)

Cross-File Deduplication (FastCDC)

What Is Content-Defined Deduplication?

Traditional compression algorithms (like LZMA, DEFLATE, and BZIP2) operate on individual files or solid stream blocks. If you archive 10 versions of a document, a folder containing duplicated libraries, or daily virtual machine disk snapshots, traditional tools compress the redundant data over and over again.

DodaZIP incorporates FastCDC (Fast Content-Defined Chunking) with cryptographic BLAKE3 hashing to achieve cross-file block deduplication. Identical data blocks—even if located inside completely different files or shifted by a few bytes—are stored only once.


⚡ How FastCDC Works

Traditional fixed-block deduplication splits files at fixed byte intervals (e.g. every 64 KB). If even a single byte is inserted at the beginning of a file, every subsequent block boundary shifts, causing 100% deduplication failure.

FastCDC avoids this problem using rolling hashes:

  1. A rolling hash scans the data byte by byte.
  2. When the hash matches specific mathematical bit-patterns, a chunk boundary is declared.
  3. Because boundaries are determined by the content rather than fixed file offsets, inserting or deleting bytes only alters the immediate chunk—all surrounding chunks retain identical boundaries and hashes.
File A: [ Chunk 1 ] [ Chunk 2 ] [ Chunk 3 ]
File B: [ Inserted Byte ] [ Chunk 1 ] [ Chunk 2 ] [ Chunk 3 ]
Result: Chunk 1, Chunk 2, and Chunk 3 are recognized as identical and stored ONCE!

🧬 Content-Addressed Chunk Table

When creating a .dzip archive, DodaZIP builds an internal index table:

  • BLAKE3 Hash: Unique 256-bit cryptographic fingerprint per chunk.
  • Reference Counting: Tracks how many files and locations utilize the same chunk.
  • Physical Offset: The exact compressed block location in the archive payload.

During extraction, DodaZIP reassembles files by streaming their corresponding chunk references in parallel.


💰 Real-World Space & Cost Savings

WorkloadTraditional ZIP/7z SizeDodaZip FastCDC SizeSpace Saved
Daily SQL Database Dumps (7 days)~35 GB~6.2 GB82%
Virtual Machine Snapshots (.vmdk / .qcow2)~120 GB~28 GB76%
Git Repositories with node_modules~4.5 GB~1.1 GB75%
Iterative Document / Backup Versions~18 GB~4.8 GB73%

💻 CLI Usage

# Create an archive with FastCDC deduplication enabled (default for .dzip)
dodazip c -i ./daily-backups -o backups.dzip --codec zstd

# Check deduplication ratio and savings summary
dodazip l backups.dzip --verbose
Stay Updated

Subscribe to Our Newsletter

Product updates, security tips, and release announcements delivered directly to your inbox.

No spam. Unsubscribe anytime.