test failures on s390x (big-endian)

[rathann]

When running version 95 tests on a big-endian machine like s390x, I get the following test error:

./check.py --binaryen-bin s390x-redhat-linux-gnu/bin
warning: no mozjs found (did not check native wasm support nor asm.js validation)
warning: no interpreter provided (did not test spec interpreter validation)
...
.. fuzz-exec_all-features.wast
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print --debug
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print --debug
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print --debug
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print --debug
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print --debug
executing:  /builddir/build/BUILD/binaryen-version_95/s390x-redhat-linux-gnu/bin/wasm-opt --fuzz-exec --all-features split.wast --print
incorrect output, diff:
--- /builddir/build/BUILD/binaryen-version_95/test/passes/fuzz-exec_all-features.txt
+++ actual
@@ -66,7 +66,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 (module
@@ -135,7 +135,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 [fuzz-exec] comparing aligned_for_size
Traceback (most recent call last):
  File "/builddir/build/BUILD/binaryen-version_95/./check.py", line 403, in <module>
    sys.exit(main())
  File "/builddir/build/BUILD/binaryen-version_95/./check.py", line 386, in main
    TEST_SUITES[test]()
  File "/builddir/build/BUILD/binaryen-version_95/scripts/test/wasm_opt.py", line 75, in test_wasm_opt
    shared.fail_if_not_identical_to_file(actual, expected_file)
  File "/builddir/build/BUILD/binaryen-version_95/scripts/test/shared.py", line 348, in fail_if_not_identical_to_file
    fail_if_not_identical(actual, f.read(), fromfile=expected_file)
  File "/builddir/build/BUILD/binaryen-version_95/scripts/test/shared.py", line 337, in fail_if_not_identical
    fail(actual, expected, fromfile=fromfile)
  File "/builddir/build/BUILD/binaryen-version_95/scripts/test/shared.py", line 332, in fail
    fail_with_error("incorrect output, diff:\n\n%s" % diff_str)
  File "/builddir/build/BUILD/binaryen-version_95/scripts/test/shared.py", line 320, in fail_with_error
    raise Exception(msg)
Exception: incorrect output, diff:
--- /builddir/build/BUILD/binaryen-version_95/test/passes/fuzz-exec_all-features.txt
+++ actual
@@ -66,7 +66,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 (module
@@ -135,7 +135,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 [fuzz-exec] comparing aligned_for_size

[rathann]

Still reproducible with 102 release.

[kripken]

Is there a way to debug something like this without big-endian hardware?

If not, at least reducing it on your side would be helpful - a tiny testcase could make this easy to figure out. The fastest way to get a reduced testcase might be to run ./scripts/fuzz_opt.py, and if it fails quickly, to follow the instructions it gives for reducing what it found.

[rathann]

@kripken You can reach out to one of Fedora s390x arch guys, Dan Horák to get SSH access to a s390x machine.
You can also emulate s390x using QEMU.
I'll follow your instructions to reduce the testcase, too. Stay tuned.

[rathann]

$ scripts/fuzz_opt.py --binaryen-bin /builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin --binaryen-lib /builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/lib64/binaryen 7749852837398145992
warning: no wabt found: expected str, bytes or os.PathLike object, not NoneType
warning: no wabt found: expected str, bytes or os.PathLike object, not NoneType
/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt --print-features /builddir/build/BUILD/binaryen-version_102/test/hello_world.wat --all-features
warning: no output file specified, not emitting output
POSSIBLE_FEATURE_OPTS: ['--disable-threads', '--disable-mutable-globals', '--disable-nontrapping-float-to-int', '--disable-simd', '--disable-bulk-memory', '--disable-sign-ext', '--disable-exception-handling', '--disable-tail-call', '--disable-reference-types', '--disable-multivalue', '--disable-gc', '--disable-memory64', '--disable-typed-function-references']
checking a single given seed 7749852837398145992

ITERATION: 1 seed: 7749852837398145992 size: 14608 (mean: 14608.0, stddev: 0.0) speed: 21076.904522613066 iters/sec,  0.0 wasm_bytes/sec

randomized pass debug: 
randomized feature opts: --all-features --disable-threads --disable-mutable-globals --disable-nontrapping-float-to-int --disable-simd --disable-bulk-memory --disable-sign-ext --disable-exception-handling --disable-tail-call --disable-reference-types --disable-multivalue --disable-gc --disable-memory64 --disable-typed-function-references
randomized settings (NaNs, OOB, legalize): False True False
randomized opts: --optimize-stack-ir --remove-unused-names --heap2local -Os --remove-unused-names --generate-stack-ir --optimize-stack-ir --reorder-locals --remove-unused-brs -fimfs=99999999

/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt input.dat -ttf -o a.wasm --denan --all-features --disable-threads --disable-mutable-globals --disable-nontrapping-float-to-int --disable-simd --disable-bulk-memory --disable-sign-ext --disable-exception-handling --disable-tail-call --disable-reference-types --disable-multivalue --disable-gc --disable-memory64 --disable-typed-function-references
pre wasm size: 2421
/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt a.wasm --all-features --disable-threads --disable-mutable-globals --disable-nontrapping-float-to-int --disable-simd --disable-bulk-memory --disable-sign-ext --disable-exception-handling --disable-tail-call --disable-reference-types --disable-multivalue --disable-gc --disable-memory64 --disable-typed-function-references --print-features
warning: no output file specified, not emitting output
[] []
/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt a.wasm -o b.wasm --optimize-stack-ir --remove-unused-names --heap2local -Os --remove-unused-names --generate-stack-ir --optimize-stack-ir --reorder-locals --remove-unused-brs -fimfs=99999999 --denan
post wasm size: 1057
running testcase handler: Asyncify
/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt a.wasm --legalize-js-interface -o async.a.wasm
/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt b.wasm --legalize-js-interface -o async.b.wasm
!
-----------------------------------------
Exception:
  File "/builddir/build/BUILD/binaryen-version_102/scripts/fuzz_opt.py", line 1213, in <module>
    total_wasm_size += test_one(raw_input_data, given_wasm)
  File "/builddir/build/BUILD/binaryen-version_102/scripts/fuzz_opt.py", line 1021, in test_one
    testcase_handler.handle_pair(input=random_input, before_wasm='a.wasm', after_wasm='b.wasm', opts=opts + FEATURE_OPTS)
  File "/builddir/build/BUILD/binaryen-version_102/scripts/fuzz_opt.py", line 869, in handle_pair
    before = fix_output(run_d8_wasm(before_wasm))
  File "/builddir/build/BUILD/binaryen-version_102/scripts/fuzz_opt.py", line 486, in run_d8_wasm
    return run_d8_js(in_binaryen('scripts', 'fuzz_shell.js'), [wasm], liftoff=liftoff)
  File "/builddir/build/BUILD/binaryen-version_102/scripts/fuzz_opt.py", line 482, in run_d8_js
    return run_vm(cmd)
  File "/builddir/build/BUILD/binaryen-version_102/scripts/fuzz_opt.py", line 446, in run_vm
    return filter_known_issues(run(cmd))
  File "/builddir/build/BUILD/binaryen-version_102/scripts/fuzz_opt.py", line 74, in run
    print(' '.join(cmd))
-----------------------------------------
!
sequence item 0: expected str instance, NoneType found
================================================================================
You found a bug! Please report it with

  seed: 7749852837398145992

and the exact version of Binaryen you found it on, plus the exact Python
version (hopefully deterministic random numbers will be identical).

You can run that testcase again with "fuzz_opt.py 7749852837398145992"

The initial wasm file used here is saved as /builddir/build/BUILD/binaryen-version_102/out/test/original.wasm

You can reduce the testcase by running this now:

||||
vvvv


/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-reduce /builddir/build/BUILD/binaryen-version_102/out/test/original.wasm '--command=bash /builddir/build/BUILD/binaryen-version_102/out/test/reduce.sh' -t /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm -w /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm


^^^^
||||

Make sure to verify by eye that the output says something like this:

At least one of the next two values should be 0:
  0
  1
The following value should be 1:
  1

(If it does not, then one possible issue is that the fuzzer fails to write a
valid binary. If so, you can print the output of the fuzzer's first command
(using -ttf / --translate-to-fuzz) in text form and run the reduction from that,
passing --text to the reducer.)

You can also read "/builddir/build/BUILD/binaryen-version_102/out/test/reduce.sh" which has been filled out for you and includes
docs and suggestions.

After reduction, the reduced file will be in /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm
================================================================================
                
(finished running seed 7749852837398145992, see error above)
$ /builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-reduce /builddir/build/BUILD/binaryen-version_102/out/test/original.wasm '--command=bash /builddir/build/BUILD/binaryen-version_102/out/test/reduce.sh' -t /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm -w /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm
|wasm-reduce
|input: /builddir/build/BUILD/binaryen-version_102/out/test/original.wasm
|test: /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm
|working: /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm
|bin dir: /builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin
|expected result:
[ProgramResult] code: 0 stdout: 
At least one of the next two values should be 0:
   0
   0
The following value should be 1:
   1
[====]
in 0.285102 seconds
[/ProgramResult]

|!! Make sure the above is what you expect! !!

|checking that command has different behavior on different inputs (this verifies that the test file is used by the command)
|checking that command has expected behavior on canonicalized (read-written) binary
|input size: 2421
|starting reduction!
|  reduce using passes...
|    command "/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm -o /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm -Oz -all" succeeded, reduced size to 920
|    command "/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm -o /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm -O3 -all" succeeded, reduced size to 863
|    command "/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm -o /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm -O4 -all" succeeded, reduced size to 811
|    command "/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm -o /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm --remove-imports -all" succeeded, reduced size to 766
|    command "/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm -o /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm --remove-memory -all" succeeded, reduced size to 738
|    command "/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm -o /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm --simplify-locals --vacuum -all" succeeded, reduced size to 734
|  after pass reduction: 734
|  pass progress: 1687, last destructive: 0
|  progress is good, do not quickly decrease factor
|  reduce destructively... (factor: 183)
|      tryToReplaceCurrent succeeded (in 0)
|      tryToReplaceCurrent succeeded (in 0)
|      tryToReplaceCurrent succeeded (in 0)
|    9% of funcs complete
|    18% of funcs complete
|      tryToReplaceCurrent succeeded (in 2)
|    27% of funcs complete
|      tryToReplaceCurrent succeeded (in 3)
|    36% of funcs complete
|    45% of funcs complete
|    54% of funcs complete
|    63% of funcs complete
|    72% of funcs complete
|      tryToReplaceCurrent succeeded (in 8)
|      tryToReplaceCurrent succeeded (in 8)
|    81% of funcs complete
|    90% of funcs complete
|    100% of funcs complete
|    try to simplify memory
|    try to remove functions
|        emptied 1 / 1 functions
|        emptied 2 / 2 functions
|        emptied 4 / 4 functions
|        emptied 1 / 1 functions
|    try to remove functions
|        removed 1 functions
|        removed 2 functions
|        removed 4 functions
|        removed 1 functions
|    try to remove functions
|        emptied 1 / 1 functions
|        emptied 1 / 1 functions
|    try to remove functions
|        removed 1 functions
|        removed 1 functions
|    try to remove functions
|        emptied 1 / 1 functions
|    try to simplify elem segments
|    try to remove exports (with factor 183)
|      removed 1 exports
|  destructive reduction led to size: 50
|  reduce using passes...
|    command "/builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm -o /builddir/build/BUILD/binaryen-version_102/out/test/t.wasm -Oz -all" succeeded, reduced size to 8
|  after pass reduction: 8
|  pass progress: 42, last destructive: 29
|  reduce destructively... (factor: 2)
|    try to simplify memory
|    try to remove functions
|    try to simplify elem segments
|    try to remove exports (with factor 2)
|  reduce destructively... (factor: 1)
|    try to simplify memory
|    try to remove functions
|    try to simplify elem segments
|    try to remove exports (with factor 1)
|  destructive reduction led to size: 8
|  reduce using passes...
|  after pass reduction: 8
|finished, final size: 8

original.wasm.gz
t.wasm.gz
w.wasm.gz

I hope this is what you need.

[kripken]

Size 8 is the smallest possible wasm file, so it's failing even to load that..?

What error do you get when you do wasm-opt w.wasm on that last wasm file?

[rathann]

No error:

$ /builddir/build/BUILDROOT/binaryen-102-1.fc34.s390x/usr/bin/wasm-opt /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm
warning: no passes specified, not doing any work
warning: no output file specified, not emitting output
$ 

[kripken]

Oh, then something went wrong with the reduction. When the reducer ran such a command, it failed for it - perhaps you are in a different directory or otherwise it can't find wasm-opt. Running the fuzzer script from the binaryen directory is what I normally do (./scripts/fuzz_opt.py) and it's possible you need to set up the PATH otherwise, or pass --binaries to the reduction script so it finds them.

You can run this to see:

./scripts/fuzz_opt.py 7749852837398145992 /builddir/build/BUILD/binaryen-version_102/out/test/w.wasm

That runs the fuzzer script with the seed that failed, and gives it the w.wasm file that the reduction arrived it. If that succeeds, try

./scripts/fuzz_opt.py 7749852837398145992

(without the file at the end, it will run the entire testcase).

[kapouer]

Note that practically all big-endian architectures have the same test failure (hppa, powerpc, ppc64, sparc64, s390x).
Is there something preventing binaryen to run on a big-endian host ?

[kripken]

I can't think of any big-endian specific issue. We don't do any low-level bit manipulation of that sort, in general. But perhaps we have undefined behavior somewhere, that's my best guess.

[rathann]

Well, this is still reproducible with 122:

incorrect output, diff:
--- /builddir/build/BUILD/binaryen-122-build/binaryen-version_122/test/passes/fuzz-exec_all-features.txt
+++ actual
@@ -66,7 +66,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 (module
@@ -135,7 +135,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 [fuzz-exec] comparing aligned_for_size
Traceback (most recent call last):
  File "/builddir/build/BUILD/binaryen-122-build/binaryen-version_122/./check.py", line 408, in <module>
    sys.exit(main())
             ~~~~^^
  File "/builddir/build/BUILD/binaryen-122-build/binaryen-version_122/./check.py", line 391, in main
    TEST_SUITES[test]()
    ~~~~~~~~~~~~~~~~~^^
  File "/builddir/build/BUILD/binaryen-122-build/binaryen-version_122/scripts/test/wasm_opt.py", line 83, in test_wasm_opt
    shared.fail_if_not_identical_to_file(actual, expected_file)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/binaryen-122-build/binaryen-version_122/scripts/test/shared.py", line 363, in fail_if_not_identical_to_file
    fail_if_not_identical(actual, f.read(), fromfile=expected_file)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/binaryen-122-build/binaryen-version_122/scripts/test/shared.py", line 352, in fail_if_not_identical
    fail(actual, expected, fromfile=fromfile)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/binaryen-122-build/binaryen-version_122/scripts/test/shared.py", line 347, in fail
    fail_with_error("incorrect output, diff:\n\n%s" % diff_str)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/binaryen-122-build/binaryen-version_122/scripts/test/shared.py", line 335, in fail_with_error
    raise Exception(msg)
Exception: incorrect output, diff:
--- /builddir/build/BUILD/binaryen-122-build/binaryen-version_122/test/passes/fuzz-exec_all-features.txt
+++ actual
@@ -66,7 +66,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 (module
@@ -135,7 +135,7 @@
 [fuzz-exec] calling unaligned_notify
 [trap unaligned atomic operation]
 [fuzz-exec] calling wrap_cmpxchg
-[LoggingExternalInterface logging 42]
+[LoggingExternalInterface logging 704643072]
 [fuzz-exec] calling oob_notify
 [trap final > memory: 18446744073709551512 > 65514]
 [fuzz-exec] comparing aligned_for_size

Note that treating the little-endian bit-representation of 42 as big-endian gives us exactly 704643072. See in Python:

>>> int(bin(42)+24*'0',2)
704643072

[tlively]

Looks like ShellExternalInterface::Memory::{set,get} do not behave correctly on big-endian architectures. They should be reversing the bytes before storing and after loading values.