nix's build is non deterministic on aarch64-darwin

[Qyriad]

Describe the bug

The derivation for Nix itself does not produce the same output every time on aarch64-darwin

Steps To Reproduce

1. clone https://github.com/NixOS/nix on an aarch64-darwin machine (reproduced most recently on ef26133)
2. run nix build
3. run nix build --rebuild
4. observe the message error: derivation '/nix/store/bihwslylwknilfw0pvwdry5nfp5snjhh-nix-2.22.0pre20240328_ef26133.drv' may not be deterministic: output '/nix/store/wsy8xc2c2s2q1na18h6n1jc0q9qjv5sp-nix-2.22.0pre20240328_ef26133' differs

Expected behavior

Nix's build should have deterministic outputs

nix-env --version output:

nix-env (Nix) 2.19.2

(The Nix version I'm using to build Nix master)

Additional context

I reproduced this with sandboxing on, and on two different M1 MacBooks.

Priorities

Add 👍 to issues you find important.

[roberth]

I suppose it didn't log something about why it's different?
It'd be good to know which files are affected, and how the differ.

[SkamDart]

I could also reproduce this on an M3 Mac, so I want to say this is specific to Darwin.

$ nix build
$ realpath ./result
/nix/store/94hzn35yg9h2m1fh46ji1czlkaz3xnvh-nix-2.22.0pre20240404_19016c7
bash-5.2$ nix build --rebuild --keep-failed
error: derivation '/nix/store/sfhsw21ja48hanisiypchzpwx28dxihg-nix-2.22.0pre20240404_19016c7.drv' may not be deterministic: output '/nix/store/94hzn35yg9h2m1fh46ji1czlkaz3xnvh-nix-2.22.0pre20240404_19016c7' differs from '/nix/store/94hzn35yg9h2m1fh46ji1czlkaz3xnvh-nix-2.22.0pre20240404_19016c7.check'

I ran these two directories against diffoscope and uploaded HTML to a branch in my fork of nix. There's significantly less noise when you ignore timestamps using the --exclude-directory-metadata=recursive option.

$ diffoscope /nix/store/94hzn35yg9h2m1fh46ji1czlkaz3xnvh-nix-2.22.0pre20240404_19016c7 /nix/store/94hzn35yg9h2m1fh46ji1czlkaz3xnvh-nix-2.22.0pre20240404_19016c7.check

See nix-build-exclude-timestamps and nix-build-diff-report from https://github.com/SkamDart/nix/tree/aarch64-darwin-reproducibility-issue if you're interested in looking at report itself.

My current hunch, though admittedly completely unverified, is the reproducibility check does not like one or both of the preConfigure or postInstall phases where we modify library names and rpaths.

nix/package.nix

Lines 268 to 285 in 24604d0

	preConfigure = lib.optionalString (doBuild && ! stdenv.hostPlatform.isStatic) (
	''
	# Copy libboost_context so we don't get all of Boost in our closure.
	# https://github.com/NixOS/nixpkgs/issues/45462
	mkdir -p $out/lib
	cp -pd ${boost}/lib/{libboost_context*,libboost_thread*,libboost_system*} $out/lib
	rm -f $out/lib/*.a
	'' + lib.optionalString stdenv.hostPlatform.isLinux ''
	chmod u+w $out/lib/*.so.*
	patchelf --set-rpath $out/lib:${stdenv.cc.cc.lib}/lib $out/lib/libboost_thread.so.*
	'' + lib.optionalString stdenv.hostPlatform.isDarwin ''
	for LIB in $out/lib/*.dylib; do
	chmod u+w $LIB
	install_name_tool -id $LIB $LIB
	install_name_tool -delete_rpath ${boost}/lib/ $LIB || true
	done
	install_name_tool -change ${boost}/lib/libboost_system.dylib $out/lib/libboost_system.dylib $out/lib/libboost_thread.dylib
	''

nix/package.nix

Lines 334 to 339 in 24604d0

	'' + lib.optionalString stdenv.isDarwin ''
	install_name_tool \
	-change ${boost}/lib/libboost_context.dylib \
	$out/lib/libboost_context.dylib \
	$out/lib/libnixutil.dylib
	''

[lf-]

we believe this may also be related, that there are non-relative paths getting in because the build dir is leaking into some debug info? NixOS/nixpkgs#299904

[IreneKnapp]

from a great deal of time spent with hex dumps, I have determined that in addition to the OSO symbols having absolute paths in them by default per lf-'s comment, they also have timestamps from the filesystem.

stdenv already sets ZERO_AR_DATE which is supposed to prevent this from happening. the environment variable name suggests that it only applies to the ar tool, but from inspecting the source of ld64 I confirmed that it is respected by the linker as well, and that nixpkgs is using a linker version new enough to have it.

this resource mentions that other distros have had success with setting the timestamps of .o files to zero in the filesystem, prior to linking. however, that could confuse the build system. I suggest that it would be more appropriate to figure out why the environment variable isn't having the effect it's supposed to.

my hex analysis also suggested that there is still another difference to account for, even after the two issues with OSO symbols. it's in the compressed part of the symbol table, and whatever it is, it doesn't result in any changes that show up in the stuff the various LLVM analysis tools know how to print out. somebody with a Mac might be able to get somewhere on that part with https://github.com/DeVaukz/MachO-Explorer