Distinguish exceptions thrown by an asserted expression vs. during expression decomposition

[eramongodb]

Given the following test case (using Catch2 v3.8.1):

#include <catch2/catch_test_macros.hpp>
#include <catch2/catch_tostring.hpp>

struct S {
  // May throw due to runtime conditions.
  void check_validity() const;

  // May throw if this object is invalid.
  std::string to_string() const { this->check_validity(); return "S"; }

  // A decomposable expression.
  friend bool operator==(S const&, S const&) { return true; }
};

// Support stringification.
template <> struct Catch::StringMaker<S> {
  static std::string convert(S const& s) { return s.to_string(); }
};

TEST_CASE("example") {
  CHECK(S() == S());
  SUCCEED("this test case should succeed");
}

// Unconditionally throws for this example.
// Real code may depend on non-trivial runtime conditions.
void S::check_validity() const { throw std::runtime_error("invalid"); }

executing the test suite with ./main produces the following outcome:

===============================================================================
All tests passed (2 assertion in 1 test case)

However, executing the test suite with ./main --success produces the following outcome (note the assertion count also increased from 2 to 3):

...............................................................................

main.cpp: PASSED:
  CHECK( S() == S() )
main.cpp: FAILED:
  CHECK( S() == S() )
due to unexpected exception with message:
  invalid

main.cpp: PASSED:
with message:
  this test case should succeed

===============================================================================
test cases: 1 | 1 failed
assertions: 3 | 2 passed | 1 failed

Furthermore, with -r compact, the resulting output is misformatted (note the missing newline):

main.cpp: passed: S() == S()main.cpp: failed: unexpected exception with message: 'invalid'; expression was: S() == S()
main.cpp: passed: with 1 message: 'this test case should succeed'
test cases: 1 | 1 failed
assertions: 3 | 2 passed | 1 failed

---

Using --success should not change the outcome of the test suite!

---

There are several possible approaches that come to mind regarding how to address this issue (these are not mutually exclusive):

• Document that --success may change the outcome of the test suite.
  • I do not like this as a final solution, but I believe it is worth documenting nevertheless.
• Document (and enforce?) that StringMaker<T> user-defined specializations or operator<< overloads should not throw an exception.
  • I do not believe this is mentioned or addressed at all by current documentation. There is no noexcept or try-catch block included in any examples I could find.
  • This solution would require adding a try-catch block within StringMaker<T>::convert() to guard against the potential exception thrown by .to_string(). This
• Ensure expression decomposition is resilient to exceptions thrown during expression decomposition by:
  • falling back to the usual "{?}" expansion if an exception was thrown, and/or
  • emitting a warning that an unexpected exception was thrown during stringification.

Ultimately, I expect/hope the test suite would behave as follows:

...............................................................................

catch2/catch_tostring.hpp: warning:
  unexpected exception during stringification of S: invalid
catch2/catch_tostring.hpp: warning:
  unexpected exception during stringification of S: invalid
main.cpp: PASSED:
  CHECK( S() == S() )
with expansion:
  {?} == {?}

main.cpp: PASSED:
with message:
  this test case should succeed

===============================================================================
All tests passed (2 assertions in 1 test case)

Notably, the test suite passes (the same outcome as without --success) despite the thrown exceptions, and the emitted warnings clearly indicate the unexpected exception was thrown during expression decomposition rather than by the expression being asserted, which did not throw an exception (as expected) and is therefore correctly reported as a pass.