Skip to content

Pre-allocate the output file on Darwin to avoid APFS sparse-file corruption#361

Merged
folbricht merged 1 commit into
masterfrom
preallocate-darwin
Jul 5, 2026
Merged

Pre-allocate the output file on Darwin to avoid APFS sparse-file corruption#361
folbricht merged 1 commit into
masterfrom
preallocate-darwin

Conversation

@folbricht

Copy link
Copy Markdown
Owner

Combines #326 by @nsakaimbo with the review fixes from nsakaimbo#1 into a single PR. Supersedes #326.

Problem

On macOS/APFS, os.Truncate creates a sparse file. When AssembleFile's concurrent workers WriteAt adjacent regions of a large sparse file, the assembled output has shown non-deterministic corruption — correct size, but null-chunk regions occasionally read back wrong (reported in #326 with ~350 GB VM disk images on M1, where forcing read-back verification made the problem disappear). Physically pre-allocating the blocks before assembly eliminates it, at no cost to assembly time.

Changes

  • AssembleFile now calls preallocateFile instead of os.Truncate for regular file targets. Block devices are unaffected.
  • Darwin (preallocate_darwin.go): uses unix.FcntlFstore with F_PREALLOCATE/F_ALLOCATEALL to physically allocate blocks, then truncates to the final size.
    • F_PEOFPOSMODE allocates relative to the current end of file, so only the missing difference (size - current) is requested. This keeps in-place reassembly of an existing file from allocating the full size again beyond EOF (which could spuriously fail with ENOSPC), and skips the fcntl entirely when the file doesn't need to grow.
    • Falls back to plain truncate on ENOTSUP — not all filesystems support F_PREALLOCATE (SMB, some FUSE mounts), and the sparse-hole issue is APFS-specific.
  • All other platforms (preallocate_other.go): delegates to Truncate, no behavior change.
  • Unlike fix: pre-allocate on Darwin to prevent APFS sparse-hole corruption #326, the 100-chunk limit in nullChunkSeed.LongestMatchWith is kept. It was added in Do not immediately abort the extraction if a seed chunk is invalid  #220 for worker load-balancing, not correctness: chunks beyond the limit are picked up by the next LongestMatchWith call or the store, and removing it would make a single worker serially zero-fill unbounded null runs during in-place assembly on non-reflink filesystems. The pre-allocation alone addresses the corruption.

Tests

  • preallocate_test.go (all platforms): new, grown (original content preserved, tail reads as zeros), shrunk, same-size, and empty files.
  • preallocate_darwin_test.go: regression test for the delta calculation — re-preallocates an existing file to its current size on a nearly-full APFS volume created with hdiutil, which fails with ENOSPC if the full size is requested instead of the difference.

One behavior change to be aware of: with F_ALLOCATEALL, extracting a mostly-zero image on macOS now requires the full physical space up front rather than succeeding sparsely. That matches the intent of the existing "confirm there's enough disk space" comment.

AssembleFile truncated the output file to its target size, which on
APFS creates a sparse file. Concurrent WriteAt calls on adjacent
regions of large sparse files have shown non-deterministic corruption
there, with null-chunk regions occasionally reading back wrong
(#326).

Replace the plain truncate with preallocateFile: on Darwin it uses
fcntl(F_PREALLOCATE) to physically allocate blocks before truncating.
Since F_PREALLOCATE with F_PEOFPOSMODE allocates relative to the
current end of file, only the missing difference is requested,
keeping in-place reassembly of existing files from over-allocating.
Filesystems without F_PREALLOCATE support (SMB, FUSE) fall back to
the previous plain-truncate behavior. All other platforms delegate
to Truncate as before.
@folbricht folbricht merged commit b258bf1 into master Jul 5, 2026
3 checks passed
@folbricht folbricht deleted the preallocate-darwin branch July 5, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant