Error during verification after step_qonnx_to_finn

[filippouxry]

Quick summary

I am trying to follow the build process using an autoencoder model trained using Pytorch in this notebook samples/onnx_accelerator_sample1/notebooks/02 - Training with Pytorch.ipynb as part of a larger project. I pass the resulting model through the Brevitas PTQ process in order to quantize to 8-bits and then feed it to the FINN flow . During the first step, step_qonnx_to_finn, I come across "Could not extract bias from node" warnings for each Conv and ConvTranspose node in the network, and then during the verification process with python after this step, the build fails with an ONNXRuntimeError.

Details

I have verified the model's functionality and accuracy after the Brevitas PTQ process.
This is the model that i feed to the FINN flow :

The intermediate model after step_qonnx_to_finn is not saved.

This is the output of the overall build:

and these are the contents of the build_dataflow.log:

this refers to the first Conv node and is repeated for the other two, and the same warning shows up for all the ConvTranspose nodes like so :

Steps to Reproduce

1. Clone the FINN repository
2. Checkout the dev branch, with commit hash: [5d51eec]
3. Start the docker container with the command.
4. Run the dataflow builder standard steps with the following config:

Expected behavior

The first step should convert the input .onnx model into a finn-dialect model suitable for further transformations and save it in the intermediate models folder. Also an output file should be saved in the verification output folder .

Actual behavior

The build fails due to the above error, not having saved the model after the qonnx_to_finn tranformation.

Optional

Possible fix

I have looked through existing issues and haven't found a definitive answer or solution, though there is mention of the same warnings ("Could not extract bias bias from node") I get in this issue comment : https://github.com/Xilinx/finn/issues/855#issuecomment-2072120388
I believe these warnings could play an important role to this functional problem down the line.

Additional context

If I skip the verification of step_qonnx_to_finn the build continues , though during the step_set_fifo_depths an error about folded shape mismatch occurred, and i had to write a custom step to alter the dilation of a ConvolutionInputGenerator layer in order for the build to finish. In this case, when testing the accelerator on a ZCU104 device, it's output was all zeros.

Any help on this would be greatly appreciated, feel free to ask for any other context needed!

UPDATE

Having by-passed verification in step_qonnx_to_finn, I managed to perform verification after my custom streamline step obtaining good enough results, but only after adding a patch in extract_conv_bias.py .

I noticed that during bias extraction the tensor_name that was looked up in the graph initializers for each Conv node was something like "Quant_2_out0" whereas the initializers included "Quant_2_param0" instead.
This is the patch I added in extract_conv_bias.py, to force the lookup of the correct (?) names:

and these are the steps I follow as well as the custom streamlining step:

Unfortunately, I can't seem to perform verification further down the process after custom_step_convert_to_hw:

because of this error:

which, I have pinpointed, refers to the MVAU of the first ConvTranspose layer. This is the model after hw conversion:

@auphelia , @hleblevec , @fpjentzsch I would really appreciate any advice on this, I think the problem could be some misalignment during a transformation (or a redundant transformation?) , possibly because the lowering of the convs happens during streamlining while the lowering of convTranspose nodes happens during conversion to hw?

[auphelia]

Hi @filippouxry,
Thank you for creating the issue and already providing an update! I will have a look into this in the next days and come back to you.

[auphelia]

Could you in the meanwhile please test with recent finn dev? We were a bit behind on the Brevitas commit but updated that now.

I am wondering about the fix you had to make for the ExtractBias transformation, the transformation is grabbing the name of the tensor from the node directly, this indicates that the bias input is not node.input[2] in your case but maybe has a different index. It might be that your onnx graph has an opset version that we didn't come across yet with FINN.

Could you instead of sharing your notebook, share your onnx model and a python script which goes through the steps?

[filippouxry]

@auphelia Thank you for the swift reply, I'll see what I can do about testing with the most recent finn dev.

I have inluded in this zip file the model after Brevitas PTQ and the custom steps script as well as the main build notebook, let me know if you need something else!

model_scripts.zip

[filippouxry]

@auphelia I have tested the whole procedure using the latest finn-dev branch and I seem to run into the exact same behavior I described, in both the case of using the patch I added and not using it.

This is part of a time-critical project me and my team are working on, so any hint you can give us would be extremely valuable !
Could you please have a look as soon as possible?
I thank you in advance.