fix: pre-allocate on Darwin to prevent APFS sparse-hole corruption

[nsakaimbo]

Summary

Fix assembly correctness on macOS/APFS when files contain large zero-filled regions (e.g., VM disk images). Two changes, no public API modifications:

1. Replace os.Truncate with preallocateFile in AssembleFile — on Darwin, uses fcntl(F_PREALLOCATE) to physically allocate blocks; on other platforms, delegates to os.Truncate (no behavior change)
2. Remove the 100-chunk limit in nullChunkSeed.LongestMatchWith (performance optimization, unnecessary since WriteInto returns immediately when isBlank=true)

Problem

On macOS/APFS, os.Truncate creates sparse files — zero regions are not physically allocated. When AssembleFile's concurrent workers call WriteAt on adjacent regions of a sparse file, APFS's copy-on-write handling causes non-deterministic data corruption. The assembled file has the correct size but different content each run.

This only affects macOS/APFS. On Linux (ext4), Truncate produces real zeros and the existing code works correctly.

Root Cause

Discovered while testing CDC delta downloads for ~350 GB VM disk images on Mac Mini M1 hosts (APFS):

• SHA-256 of assembled output didn't match the source — non-deterministic (different wrong hash each run)
• cmp against the source found the first divergence at a chunk boundary where the assembled file had zeros instead of data
• Forcing isBlank=false (read-back verification on every chunk) produced correct output, confirming the issue is in the isBlank=true path interacting with APFS sparse holes
• fcntl(F_PREALLOCATE) before truncation physically allocates blocks, eliminating sparse holes — isBlank=true then works correctly at full speed

The bug only manifests with very large files (~350 GB with ~44 GB of zero regions). Small test files don't reproduce it.

Fix

preallocateFile (preallocate_darwin.go): Calls fcntl(F_PREALLOCATE) + ftruncate to allocate real blocks. The isBlank skip in the null seed's WriteInto remains correct because the pre-allocated blocks already contain zeros.

preallocateFile (preallocate_other.go): Delegates to os.Truncate — no behavior change on Linux or other platforms.

Performance

Tested with a 325 GB VM disk image (46,638 chunks, 2,853 null chunks, max run of 1,942 consecutive) on Mac Mini M1:

Approach	Assembly Time	SHA-256
Unpatched (APFS sparse)	7m 20s	❌ Non-deterministic
Force isBlank=false	10m 38s	✅
This PR	7m 20s	✅

No performance impact on Linux.

[folbricht]

Thank you. Good catch.

Though can you check if the 2nd change is actually necessary. That's quite the perf impact.

	if !s.canReflink {
		if isBlank {
			return 0, 0, nil
		}

This suggests that truncating an empty file does not end up with an empty file. You can try it with the new test, it still passes without that change.

[folbricht]

Interesting, the new test doesn't fail with the old code, at least on Linux. Would be nice to figure out what the actual issue is first.

[folbricht]

@nsakaimbo I opened nsakaimbo#1 against your PR with what I think the fixes should look like. Please take a look and merge into yours if everything looks ok.

[folbricht]

I spent some time trying to get to the root cause of this before merging a fix, since the corruption mechanism was never really established. Short version: I was unable to reproduce it, despite trying under conditions considerably more hostile than the original report, and a code audit found no desync-side mechanism that could explain the symptom. I'm going to merge #361 anyway — the preallocation is cheap (your own numbers showed no assembly-time cost), and it removes the dependency on sparse-hole behavior entirely, so it works around the issue regardless of where it lives.

What was tried, on an Apple Silicon machine running macOS 26.5.2 / Darwin 25.5.0:

• Stress-testing the exact I/O pattern outside desync — sparse-truncated output files, concurrent workers with private file descriptors doing non-overlapping WriteAt, large never-written hole runs abutting data, concurrent hole read-back (matching the seed-validation loop), and read-from-output copies (matching the self-seed path). 143 runs across 20–110 GiB files, default and ~8 MiB chunk geometry, with heavy memory pressure forcing continuous eviction and writeback: all clean. Warm-cache and cold-cache (unmount/remount) hashes matched every time, so nothing was lost at writeback either. The verification logic was itself validated by fault injection, so the clean results aren't a detector blind spot.
• The actual pre-fix desync binary (master before this PR), extracting synthetic workloads matching your index's geometry (~7 MiB average chunks, multi-GiB null runs, ~40k duplicate chunks to exercise the self-seed), under memory pressure: 22/22 extracts, bit-identical correct SHA-256.
• A code audit looking for any way desync could silently produce zeros where data belonged: store and cache chunks are hash-verified before writing (unless --skip-verify), cache writes are atomic renames, the self-seed only offers ranges whose writes have completed, and seed segments — including skipped null segments — are read back and hash-verified during assembly. Nothing found.

So if this is real — and the non-deterministic, silent, zeros-for-data symptom does look like lost writes — it's most likely specific to your environment: macOS build, M1-generation hardware, the target volume, or genuine ≥350 GB scale, none of which I could replicate (the test machine caps out around 110 GiB usable).

If you're still able to observe this, a few things would help narrow it down:

1. sw_vers and uname -a from the affected machine.
2. diskutil info for the volume you extracted to — was it the internal drive or an external one?
3. Was the store remote at extract time (chunks arriving slowly over the network), and was --skip-verify in use?

Merging #361 now. Thanks again for the report and the detailed measurements.