-
-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Description
Describe the bug
JUnit reporter can emit multiple failure entries in the case of multiple CHECK failures. This does not conform to the JUnit specification: http://junit.sourceforge.net/doc/faq/faq.htm#tests_12
Why does JUnit only report the first failure in a single test?
Reporting multiple failures in a single test is generally a sign that the test does too much, compared to what a unit test ought to do.
As a result, some tools that might attempt to parse the JUnit output from Catch2 will fail. I am seeing https://github.com/gastlygem/junitparser/blob/master/junitparser/junitparser.py fail with raise JUnitXmlError("Only one result allowed per test case.").
Expected behavior
To conform to the behavior of JUnit, only the first failure should be output.
Reproduction steps
#define CATCH_CONFIG_MAIN
#include "catch.hpp"
TEST_CASE("errors")
{
CHECK(false);
CHECK(false);
}
launch with -r junit -o junit.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="Project1.exe" errors="0" failures="2" tests="2" hostname="tbd" time="0.001193" timestamp="2020-04-29T00:00:20Z">
<testcase classname="Project1.exe.global" name="errors" time="0.000377">
<failure message="false" type="CHECK">
FAILED:
CHECK( false )
at e:\dev\catch2junitbug\project1\main.cpp(6)
</failure>
<failure message="false" type="CHECK">
FAILED:
CHECK( false )
at e:\dev\catch2junitbug\project1\main.cpp(7)
</failure>
</testcase>
<system-out/>
<system-err/>
</testsuite>
</testsuites>
Platform information:
- OS: Windows 10
- Compiler+version: MSVC toolchain v141
- Catch version: 2.12.1