Test Report from OpenDAL

[Xuanwo]

NOTE: This test report is for informational purposes only and does not suggest any immediate need for repair.

Following the instructions at #25, I attempted to use rustc_codegen_clr with opendal. OpenDAL is a REAL Rust crate that facilitates IO, networking, and asynchronous operations. If we successfully compile OpenDAL and integrate it with CLR, rustc_codegen_clr will be nearing maturity.

• Test upon: https://github.com/Xuanwo/rustc_codegen_clr/tree/opendal-test
• Test with: cargo test compile_test::opendal_test::cargo_release
• Test build log: build_log.txt.gz

Worth of mention:

query stack during panic:
end of query stack
Method "_ZN10proc_macro6bridge6client10run_client28__ds_u7b_ds__ds_u7b_ds_closure_ds_u7d_ds__ds_u7d_ds_17h4767fd0569f4f2f7E" failed to compile statement _41 = <proc_macro::bridge::buffer::Buffer as std::convert::From<std::vec::Vec<u8>>>::from::reserve as extern "C" fn(proc_macro::bridge::buffer::Buffer, usize) -> proc_macro::bridge::buffer::Buffer (PointerCoercion(ReifyFnPointer)) with message Error("statement_to_ops panicked with a non-string message!")
thread 'rustc' panicked at src/rvalue.rs:350:44:
not yet implemented: Unhandled cast kind PointerCoercion(ReifyFnPointer), rvalue:<proc_macro::bridge::buffer::Buffer as std::convert::From<std::vec::Vec<u8>>>::from::drop as extern "C" fn(proc_macro::bridge::buffer::Buffer) (PointerCoercion(ReifyFnPointer))
stack backtrace:
   0:     0x7f87b5b8b756 - std::backtrace_rs::backtrace::libunwind::trace::h0c778a97139c185f
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0x7f87b5b8b756 - std::backtrace_rs::backtrace::trace_unsynchronized::h49275f7f11d03db8
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f87b5b8b756 - std::sys_common::backtrace::_print_fmt::hc6d2af0f9113f87c
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x7f87b5b8b756 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h4269025391cee804
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7f87b5bde080 - core::fmt::rt::Argument::fmt::h47e13179bda442d0
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/fmt/rt.rs:142:9
   5:     0x7f87b5bde080 - core::fmt::write::h0ea06241e192b662
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/fmt/mod.rs:1120:17
   6:     0x7f87b5b7f40f - std::io::Write::write_fmt::hada0a89348d2eeaf
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/io/mod.rs:1810:15
   7:     0x7f87b5b8b534 - std::sys_common::backtrace::_print::h474b180b0f0cbfaa
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7f87b5b8b534 - std::sys_common::backtrace::print::h292cdb95c9f3ed37
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7f87b5b8e2c7 - std::panicking::default_hook::{{closure}}::h215c742b61103c6e
  10:     0x7f87b5b8e029 - std::panicking::default_hook::h57e29db39f9afc40
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:292:9
  11:     0x7f87b895895c - std[8ca417bb7e88f110]::panicking::update_hook::<alloc[e6bcedfe09e584e0]::boxed::Box<rustc_driver_impl[f66e2583066ca4e1]::install_ice_hook::{closure#0}>>::{closure#0}
  12:     0x7f87b5b8ea16 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::ha301741326637587
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2029:9
  13:     0x7f87b5b8ea16 - std::panicking::rust_panic_with_hook::hdec91a2b4b229c0a
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:783:13
  14:     0x7f87b5b8e762 - std::panicking::begin_panic_handler::{{closure}}::h307c36ea28011ec5
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:657:13
  15:     0x7f87b5b8bc56 - std::sys_common::backtrace::__rust_end_short_backtrace::h4a40c8d6f1eb7696
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:171:18
  16:     0x7f87b5b8e4c0 - rust_begin_unwind
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:645:5
  17:     0x7f87b5bda785 - core::panicking::panic_fmt::h5740899885667996
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panicking.rs:72:14
  18:     0x7f87ac5836ff - rustc_codegen_clr::rvalue::handle_rvalue::hf672d95e8333dd3d
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/rvalue.rs:350:44
  19:     0x7f87ac589930 - rustc_codegen_clr::statement::handle_statement::hcdcf07e41ea546fb
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/statement.rs:33:79
  20:     0x7f87ac55a847 - rustc_codegen_clr::assembly::Assembly::statement_to_ops::{{closure}}::h9273593ca2d4ba8d
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/assembly.rs:187:17
  21:     0x7f87ac4cc661 - core::ops::function::FnOnce::call_once::hafd45deffa031da8
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/ops/function.rs:250:5
  22:     0x7f87ac4adf1d - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h391e0a50ebfed153
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panic/unwind_safe.rs:272:9
  23:     0x7f87ac509945 - std::panicking::try::do_call::h02001182046b80cd
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:552:40
  24:     0x7f87ac509dfb - __rust_try
  25:     0x7f87ac509839 - std::panicking::try::h902cf4314083380a
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:516:19
  26:     0x7f87ac5c434e - std::panic::catch_unwind::hf726a4f904c10269
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panic.rs:142:14
  27:     0x7f87ac4c568d - rustc_codegen_clr::assembly::Assembly::statement_to_ops::h7fd31949f3c13db2
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/assembly.rs:186:19
  28:     0x7f87ac4c7145 - rustc_codegen_clr::assembly::Assembly::add_fn::h3db5633ea0057a47
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/assembly.rs:286:43
  29:     0x7f87ac55a8c0 - rustc_codegen_clr::assembly::Assembly::checked_add_fn::{{closure}}::h2962be4d0e46f30e
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/assembly.rs:211:13
  30:     0x7f87ac4cbf51 - core::ops::function::FnOnce::call_once::h2a1cbc610b6b0ba7
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/ops/function.rs:250:5
  31:     0x7f87ac4adf5b - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h590d7ee442373ebe
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panic/unwind_safe.rs:272:9
  32:     0x7f87ac509a9d - std::panicking::try::do_call::hf417cd1a732ed325
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:552:40
  33:     0x7f87ac509dfb - __rust_try
  34:     0x7f87ac509488 - std::panicking::try::h1c79b5996f62a78d
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:516:19
  35:     0x7f87ac5c432e - std::panic::catch_unwind::hb2e3c91a00650d1b
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panic.rs:142:14
  36:     0x7f87ac4c59ba - rustc_codegen_clr::assembly::Assembly::checked_add_fn::h91cbdbce40b02476
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/assembly.rs:210:15
  37:     0x7f87ac4c8cc5 - rustc_codegen_clr::assembly::Assembly::add_item::h5be52e6cc5fc8bbd
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/assembly.rs:473:17
  38:     0x7f87ac509095 - <rustc_codegen_clr::MyBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate::h8f608a38b9f77cc4
                               at /home/xuanwo/Code/FractalFir/rustc_codegen_clr/src/lib.rs:200:21
  39:     0x7f87babb502c - rustc_interface[1c04995cfe76341a]::passes::start_codegen
  40:     0x7f87babb47a6 - <rustc_interface[1c04995cfe76341a]::queries::Queries>::codegen_and_build_linker
  41:     0x7f87baa1e4e8 - rustc_interface[1c04995cfe76341a]::interface::run_compiler::<core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>, rustc_driver_impl[f66e2583066ca4e1]::run_compiler::{closure#0}>::{closure#0}
  42:     0x7f87babca9c6 - std[8ca417bb7e88f110]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[1c04995cfe76341a]::util::run_in_thread_with_globals<rustc_interface[1c04995cfe76341a]::util::run_in_thread_pool_with_globals<rustc_interface[1c04995cfe76341a]::interface::run_compiler<core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>, rustc_driver_impl[f66e2583066ca4e1]::run_compiler::{closure#0}>::{closure#0}, core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>>::{closure#0}, core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>>
  43:     0x7f87babca7f3 - <<std[8ca417bb7e88f110]::thread::Builder>::spawn_unchecked_<rustc_interface[1c04995cfe76341a]::util::run_in_thread_with_globals<rustc_interface[1c04995cfe76341a]::util::run_in_thread_pool_with_globals<rustc_interface[1c04995cfe76341a]::interface::run_compiler<core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>, rustc_driver_impl[f66e2583066ca4e1]::run_compiler::{closure#0}>::{closure#0}, core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>>::{closure#0}, core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[6d340633ba3d0ca7]::result::Result<(), rustc_span[9f22fee959df9e59]::ErrorGuaranteed>>::{closure#1} as core[6d340633ba3d0ca7]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  44:     0x7f87b5b98945 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc049aa84baee5c22
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9
  45:     0x7f87b5b98945 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hf5846f37b8112bb2
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9
  46:     0x7f87b5b98945 - std::sys::unix::thread::Thread::new::thread_start::hd01330a228cdd35d
                               at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys/unix/thread.rs:108:17
  47:     0x7f87b59849eb - <unknown>
  48:     0x7f87b5a087cc - <unknown>
  49:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

[FractalFir]

Trying to build OpenDAL seems like an interesting test.
It is quite complex, but it is also a potential real use case (since there seem to be no C# bindings for it project).

Looking at the log reveals some interesting things.

Currently, there are 2244 codegen panics in the build log.
A large chunk of them (573, ~25%) are related to calling/creating function pointers.
Another large group are the errors related to DSTs (995, ~44%).

Those 2 groups total in 1568 panics(70%). Other issues cause much fewer errors, 676(~30%).

While I am almost certain there are other issues which are not currently picked up, this suggests that a lot of progress can
be made by focusing on those areas.

ReifyFnPointer

The issue with ReifyFnPointer was also present when building core, so I fixed it.

d2d4ca1 allows creation of function pointers (but functions with alternative calling conventions / marked with the #[track_caller] attribute don't work properly yet).

5946c83 adds the ability to call those function pointers.

There are some function pointer related things that still need some consideration.

For example, distinguishing between managed and unmanged function pointers has huge performance benefits, but could potentially introduce runtime issues if not done right.

But even with those minor issues unresolved, calling/creating function pointers should work now, reducing the amount of codegen panics.

DSTs

Those also cause trouble in core, so they were going to be fixed anyaway

DSTs are a bit more tricky, since they require revamping some of the code handling fat pointers. The newest set of patches greatly improves support for DSTs, and a large chunk of them should now work, but there are still some issues.

Due to Ty::is_sized sometimes panicking, the codegen is currently using is_trivialy_sized. This may lead it to assume a type is a DST when it is not. Once I figure out the cause of the issue with is_sized, this should be working properly.

Getting the address of a dynamically sized field of a DST currently should copy the address and metadata of the field, but this currently does not happen, creating fat pointers with garbage addresses/metadata, causing runtime issues.

Once those issues get resolved, a large chunk of errors should disappear. Admittedly, those are probably the low-hanging fruit, but this should be a good step forward.

Maturity

This is a small clarification about where the project is headed.

Also, I would not say that the project will be close to being mature when it is capable of properly building OpenDAL.
I believe it would be ready to be used at that stage, but there are some other things I would like to do before it is "mature" to me.

Mainly:

1. Fuzzing the codegen using rustlatis. This would catch most codegen bugs and miscompilations, allowing the project to be used with confidence in its output.

2. Not relying on ILASM. While ilasm is great for development, it increases compile times and requires users to install additional dependencies. Dropping it would probably be a requirement for the project to become a part of the Rust compiler.

3. General fix ups/ refactoring, removing all transmute calls. Most of those things are fine in development, but will should be fixed to allow easier maintenance in the future. The codegen should not rely on implementation details to work.

4. Getting an official target triple from the Rust team - this would allow crates to make .NET specific accommodations/fixes. It is required to get some low-level crates to work.

5. Stable interop APIs

6. .NET specific fixes to the standard library. Currently, the codegen works by pretending that the compiled executable is running on a Linux machine. Things like standard output are emulated. By using .NET APIs directly, much greater compatibility and performance can be achieved.

7. Run crater - crater is an internal Rust tool used to check if/how many crates are impacted by a change to the compiler. Ideally, I would like a large part of the Rust ecosystem to work out of the box.

8. Minor management-related things. A proper code of conduct, CONTRIBUTING.md, bug report forms - things the project will need if it is used by a larger amount of people.

I know those are just nitpicky semantics (and I hope I don't come off as rude), but I am rather cautious when it comes to those things.

I would like the project to be in a near-prefect state before calling it mature.

I don't want to claim the project is more ready than it is - I would not want someone to assume it is fully working, only to surprise them with hidden issues.