[Ruby/All] Changes between 21.2 - 21.3.1 causing issues in default html report

[luke-hill]

👓 What did you see?

In 21.2.0 the report contains by default all unexecuted test cases as grey pending / undefined
In 21.2.0 the report contains a full stacktrace in Ruby

In 21.3.1 the report does not contain any pending / undefined
In 21.3.1 the report contains only the first line of the stacktrace in Ruby

✅ What did you expect to see?

In 21.2.0 the report contains by default all unexecuted test cases as grey pending / undefined
In 21.2.0 the report contains a full stacktrace in Ruby

In 21.3.1+ the report contains by default all unexecuted test cases as grey pending / undefined
In 21.3.1+ the report contains a full stacktrace in Ruby

📦 Which tool/library version are you using?

Cucumber Ruby 9.2 and various versions of html-formatter

🔬 How could we reproduce it?

Use versions above, have undefined / failing tests

📚 Any additional context?

No response

[luke-hill]

@davidjgoss mentioned this a while back. Given we have had a few more versions I figured I/we should try establish where the issue lies and what can be done if anything.

[davidjgoss]

Re undefined and pending:

In 21.2.0 the report contains by default all unexecuted test cases as grey pending / undefined

and then:

In 21.3.1 the report does not contain any pending / undefined

I can't quite tell what you're saying is happening and is wrong here?

Pending and undefined seem to work correctly for me based on a run from cucumber-js:

But then "contains by default all unexecuted test cases as grey" sounds different to that again? We did make a change to stop pickles with no test cases being included (where e.g. they were compiled but then filtered out based on tags) but this was many versions ago.

[davidjgoss]

Re stack traces, the upstream change was cucumber/react-components#345.

TLDR, if there is an exception present on a test step result, it will use that to render instead of message at the same level. That exception object has fields for type, message and stackTrace - so perhaps cucumber-ruby is not populating the latter?

[luke-hill]

I'll add screenshots for the grey difference.

As for the upstream change for messages, again I'm not saying something is necessarily broken here, but I might need some help "looking" for the bit to fix in ruby.

Not sure when I'll get to this.