Implement Record/Replay functionality for btest

[rexim]

[rexim]

I'm gonna wait 1 day for the feedback and then merge it

[deniska]

Two potential bikeshedding points:

Do we expect different outputs for different architectures? If we do, that can potentially lead to some targets having a recorded "semi-broken" behavior which will look green in the test runner, and you wouldn't know it without a deeper inspection ("vibe check") of the particular test output. If the test is expected to be broken on the particular architecture (especially runtime broken, rather than build time broken), it should be marked more clearly that this particular architecture for this particular test is "special".

And if we go with "different test output per target", perhaps put them in different files, so that diffs fixing codegen don't conflict with each other?

[Miezekatze64]

For me the ./build/btest -c asm_func_fasm_x86_64_linux -t fasm-x86_64-windows test gives the correct output on wine, although it is expected to give an incorrect output (which makes sense, because the test is linux-only, but the test system currently reports this as unexpected)

It seems like the JSON currently contains

"fasm-x86_64-windows": {
    "status": "RunSuccess",
    "stdout": "8070071\r\n"
  }

although for some reason it works correctly for me (wine version: wine-10.9).
(exact same behaviour for asm_func_gas_x86_64_linux on gas-x86_64-windows)

Maybe we could introduce a way to say that particular test should only be tested for some architectures instead of asserting failure?

[rexim]

@deniska I think pretty-printing the json files makes things less conflict and error prone c919af6 What do you think?

[deniska]

Pretty printing json makes the diff problem quite less severe, sure.

But there's still the conceptual problem of "by default we expect every architecture to have a different output".

My idea for a system like this would be to have toupper.json with the output it's supposed to have, for all targets, and then to have toupper.uxn.json if for some reason the uxn target can't do the usual output on the toupper test, with the overrides (maybe different output, or maybe it doesn't compile at all, whatever it does, reflected in the overridden json file).

[nullnominal]

LGTM Other then tests/ being bloated with the JSON files.

We think it would be a good idea to have them in a sub-folder.

[rexim]

For me the ./build/btest -c asm_func_fasm_x86_64_linux -t fasm-x86_64-windows test gives the correct output on wine, although it is expected to give an incorrect output (which makes sense, because the test is linux-only, but the test system currently reports this as unexpected)

It seems like the JSON currently contains

"fasm-x86_64-windows": {
    "status": "RunSuccess",
    "stdout": "8070071\r\n"
  }

although for some reason it works correctly for me (wine version: wine-10.9). (exact same behaviour for asm_func_gas_x86_64_linux on gas-x86_64-windows)

Maybe we could introduce a way to say that particular test should only be tested for some architectures instead of asserting failure?

@Miezekatze64 Thank you for your feedback! I changed up the concept at 4b79ae0. Now build and runtime failures are not something that we expect and you can also disable tests on certain targets right in the json. Let me know what you think.

[nullnominal]

LGTM Other then tests/ being bloated with the JSON files.

We think it would be a good idea to have them in a sub-folder.

Created a PR for this

[Miezekatze64]

For me the ./build/btest -c asm_func_fasm_x86_64_linux -t fasm-x86_64-windows test gives the correct output on wine, although it is expected to give an incorrect output (which makes sense, because the test is linux-only, but the test system currently reports this as unexpected)

It seems like the JSON currently contains

"fasm-x86_64-windows": {
    "status": "RunSuccess",
    "stdout": "8070071\r\n"
  }

although for some reason it works correctly for me (wine version: wine-10.9). (exact same behaviour for asm_func_gas_x86_64_linux on gas-x86_64-windows)
Maybe we could introduce a way to say that particular test should only be tested for some architectures instead of asserting failure?

@Miezekatze64 Thank you for your feedback! I changed up the concept at 4b79ae0. Now build and runtime failures are not something that we expect and you can also disable tests on certain targets right in the json. Let me know what you think.

Looks good now.

[rexim]

LGTM Other then tests/ being bloated with the JSON files.

We think it would be a good idea to have them in a sub-folder.

@nullnominal merged them all into a single file at b48c7e2. Let me know what you think.

[nullnominal]

LGTM Other then tests/ being bloated with the JSON files.
We think it would be a good idea to have them in a sub-folder.

@nullnominal merged them all into a single file at b48c7e2. Let me know what you think.

yeah its good

[rexim]

Thank you everybody for the feedback! Sorry that it took so long. This is a very important for me functionality which I can't continue merging other PRs without. Thank you again!