Skip to content

JUnit reporter allows multiple failures per test case #1919

@gbx-bfriedman

Description

@gbx-bfriedman

Describe the bug
JUnit reporter can emit multiple failure entries in the case of multiple CHECK failures. This does not conform to the JUnit specification: http://junit.sourceforge.net/doc/faq/faq.htm#tests_12

Why does JUnit only report the first failure in a single test?
Reporting multiple failures in a single test is generally a sign that the test does too much, compared to what a unit test ought to do.

As a result, some tools that might attempt to parse the JUnit output from Catch2 will fail. I am seeing https://github.com/gastlygem/junitparser/blob/master/junitparser/junitparser.py fail with raise JUnitXmlError("Only one result allowed per test case.").

Expected behavior
To conform to the behavior of JUnit, only the first failure should be output.

Reproduction steps

#define CATCH_CONFIG_MAIN
#include "catch.hpp"

TEST_CASE("errors")
{
	CHECK(false);
	CHECK(false);
}

launch with -r junit -o junit.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="Project1.exe" errors="0" failures="2" tests="2" hostname="tbd" time="0.001193" timestamp="2020-04-29T00:00:20Z">
    <testcase classname="Project1.exe.global" name="errors" time="0.000377">
      <failure message="false" type="CHECK">
FAILED:
  CHECK( false )
at e:\dev\catch2junitbug\project1\main.cpp(6)
      </failure>
      <failure message="false" type="CHECK">
FAILED:
  CHECK( false )
at e:\dev\catch2junitbug\project1\main.cpp(7)
      </failure>
    </testcase>
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>

Platform information:

  • OS: Windows 10
  • Compiler+version: MSVC toolchain v141
  • Catch version: 2.12.1

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions