Private writable delta
One attempt-local upperdir
Back to roadmap overview
02 · Content-defined chunking
Reusable durable deltas. Native execution stays native.
Content-defined chunking gives changed file content stable identities below LayerStack’s immutable B/L/S layers. It reduces duplicate payload while keeping the command path on ordinary native OverlayFS.
- Status
- Target design
- Scope
- Durable delta storage
- Command path
- Zero CAS lookups
- Execution
- Native OCI/Linux
The storage contract
Identity follows content; execution follows the kernel.
A sealed checkpoint records a logical tree and immutable delta layers. Within changed regular-file extents, a content-defined chunker derives stable object identities. Those identities make reuse, verification, retention, and later backend projections explicit without replacing OverlayFS with a userspace filesystem.
Capture-to-execution path
Chunk only while creating or maintaining durable state.
Capture reads the quiesced private upperdir, not the merged workspace. The resulting delta can be materialized as an ordinary immutable native lowerdir. A command then reads that lowerdir through OverlayFS with no pack, manifest, or index lookup on the normal command path.
Capture plane
Sparse extent scan
Changed allocated bytes only
Content-defined boundaries
Stable across reusable content
Immutable content objects
Typed identity · verified carrier
Durable state plane
File-segment map
Offsets, holes, metadata
Immutable L / S delta
Native B/L/S-compatible layer
LayerStack manifest
Newest first · leased root
Native lowerdir projection
Kernel OverlayFS workspace
Command / PTY
Space optimization
Near one-copy payload storage without a second execution filesystem.
The native immutable layer that OverlayFS needs is the normal physical carrier for dense file content. Packed storage is reserved for metadata, small objects, and payload that has no surviving native carrier. The index maps stable identity to verified locations rather than turning every chunk into a permanent duplicate file.
Unchanged lower payload
0 added bytes / hot fork
A child leases an existing immutable root; it does not clone it.
New payload
Changes only
A private upperdir grows with real writes; a sealed layer stores newly unique content.
Metadata
Packed + bounded
Manifests, paths, blame, journals, and small objects avoid per-object inode overhead.
Reuse
Content identity
A changed file can retain reusable content when its boundaries remain stable.
Zero additional lower payload
Per hot local fork, not zero total storage. The parent lower root still exists and remains pinned by a lease.
No permanent native + CAS double copy
A native extent is normally the durable carrier; the CAS index records its location and integrity receipt.
No full-tree clone
Publication captures the private upperdir delta. It never persists a merged workspace checkout as the durable delta.
No universal physical-space promise yet
Allocation, reflink behavior, sparse support, and OverlayFS semantics must be measured on the actual Docker-backed Linux storage domain.
Asymptotic accounting
Pay for the changed content—not the complete workspace.
Big-O describes the logical/storage work. Kernel mount setup, filesystem latency, cache state, lower-chain depth, and Docker storage capabilities remain measurable constants or separate gates; they are not erased by the notation.
Symbols
- L
- immutable lower-layer depth
- P
- changed filesystem entries
- Δ
- scanned changed payload bytes
- F
- baseline whole-file payload bytes
- U
- new unique payload bytes
- C
- new chunk / segment references
- R
- missing cold-root payload bytes
| Operation / cost | Current whole-file delta | Target CDC delta | What changes |
|---|---|---|---|
| Create / fork a logical child | Time O(1) · Space O(1) | Time O(1) · Space O(1) | No asymptotic change. Both add a NodeId, root reference, lease, and provenance—not a lower-payload copy. |
| Activate a hot local child | Time O(L) · Space O(1) | Time O(L) · Space O(1) | No asymptotic change. Both assemble a lowerdir chain and private runtime state. |
| Capture + seal changed files | Time O(F + P) · Space O(F + P) | Time O(Δ + P) · Space O(U + C + P) | The baseline retains every byte of each changed whole file. CDC retains only new content plus segment and filesystem metadata. |
| Cold hydration | Time O(R) · Space O(R) | Time O(R) · Space O(R) | No asymptotic change. A missing native root must be materialized once before the worker becomes hot. |
| N sequential revisions | Space O(ΣFᵢ) | Space O(ΣUᵢ + ΣCᵢ + ΣPᵢ) | Stable boundaries can retain unchanged regions across versions. |
| Worst-case changed content | Space O(F) | Space O(F) | When every byte is new or boundaries cannot be reused, CDC has no asymptotic space advantage. |
| Command / PTY I/O | Native OverlayFS path | Native OverlayFS path | No CAS reconstruction on either command hot path. |
Current-baseline comparison
CDC changes data-sensitive storage—not the kernel execution contract.
The current baseline captures a changed regular file as a whole immutable native-layer payload. It already keeps command execution on native OverlayFS and avoids a full merged-tree clone on a hot session. The CDC target changes the durable payload representation so an edited file can retain content that did not actually change.
Current whole-file delta
- 1
file v1·full native payload
- 2
file v2·another full payload
- 3
file v3·another full payload
Target CDC delta
- 1
chunk A·reused
- 2
chunk B′·new
- 3
chunk C·reused
Research lanes
One durable shape; four ways to decide reusable content boundaries.
All four lanes preserve the same LayerStack B/L/S and native-execution contract. L1 isolates the packed-CAS layout; L2–L4 vary the content-defined boundary algorithm. Their final storage and time behavior must be proven by the same containerized measurement gates.
- L1
Paged Footer-Pack CAS
Controls the hybrid native-carrier / packed-metadata storage layout with frozen selective FastCDC boundaries.
- L2
SeqCDC
Tests sequential content-defined boundaries and their reuse/CPU trade-off.
- L3
Gear64 SIMD
Tests exact Gear64 boundaries with scalar/SIMD equivalence requirements.
- L4
VectorCDC VAE
Tests vectorized asymmetric-extremum boundaries under the same storage contract.
Boundary of the optimization
The durable model is tree/file-aware; chunking is an internal payload codec.
The product-level root is a logical manifest with paths, file objects, directories, links, metadata, and provenance. Chunking can improve physical deduplication inside file content, but it never changes the meaning of a SandboxNode, grants mount privilege, or makes the target image responsible for storage.