Optimize buffer handling for unary ops #9607

Job	Run time
build-and-test / cmake-linux-aarch64	2m 10s
build-and-test / cmake-linux-x86_64	14m 37s
build-and-test / cmake-windows-x64	4m 17s
build-and-test / cmake-linux-riscv64	5m 59s
build-and-test / cmake-linux-riscv64-rvv	5m 33s
build-and-test / cmake-android (arm64)	1m 34s
build-and-test / cmake-windows-arm64	3m 21s
build-and-test / cmake-macos-arm64	2m 27s
build-and-test / cmake-linux-armhf	1m 45s
build-and-test / cmake-windows-x86	4m 42s
build-and-test / cmake-android (armv7)	1m 10s
build-and-test / cmake-android (x86)	1m 42s
build-and-test / cmake-ios-arm64	4m 6s
build-and-test / bazel-linux-x86_64-gcc-9	3m 2s
build-and-test / cmake-ios-x86_64	5m 12s
build-and-test / bazel-linux-aarch64-clang18	4m 6s
build-and-test / bazel-linux-x86_64-clang-18	3m 34s
build-and-test / bazel-linux-aarch64-sme2-qemu	3m 2s
build-and-test / bazel-linux-aarch64-gcc13	3m 33s
	1h 15m 52s

Provide feedback