Strange TCA Test Behavior with SnapshotTesting

[bradhowes]

In my AUv3Controls package I have some tests for the reducers and they seem to be running fine. I tried adding snapshot testing for view renders, and that ended up working OK. However, the strange part is if I run all tests (⌘U) then I get test failures in ONE
previously-working render test, but there is no info about the failure in the code editor, just a red test marker.

In the console, I see:

Test Case '-[AUv3ControlsTests.ToggleViewTests testToggleObservations]' started.
/Users/howes/Library/Developer/Xcode/DerivedData/AUv3Controls-fohqdmqmylykbhfntyvemjuhhynn/SourcePackages/checkouts/xctest-dynamic-overlay/Sources/XCTestDynamicOverlay/XCTFail.swift:37: error: -[AUv3ControlsTests.ToggleViewTests testToggleObservations] : Unimplemented: ContinuousClock.now
/Users/howes/Library/Developer/Xcode/DerivedData/AUv3Controls-fohqdmqmylykbhfntyvemjuhhynn/SourcePackages/checkouts/xctest-dynamic-overlay/Sources/XCTestDynamicOverlay/XCTFail.swift:37: error: -[AUv3ControlsTests.ToggleViewTests testToggleObservations] : Unimplemented: ContinuousClock.sleep
value: 1.0
terminating AUParameter observer
Test Case '-[AUv3ControlsTests.ToggleViewTests testToggleObservations]' failed (0.259 seconds).

But again the tests run just fine on their own, as an ensemble in the test class, OR if I comment out the
sole snapshot test.
Again I am only seeing this for one test, not others. Very perplexing.

Any ideas?

[mbrandonw]

Hi @bradhowes, the test failure is that an unimplemented clock is being accessed. Most likely it is being accessed in some effect or other escaping context that runs after the test finishes, which causes it to bleed over in other tests. And that explains why running the test by itself does not fail, because the failure is happening after the test finishes and so is not caught by the test runner. You will need to track down exactly which test is accessing this dependency and override it.

Note that these kinds of failures don't typically happen with TestStore (but it still is possible) because the test store forces all effects to finish before the test finishes. That helps properly attribute any test failures to the actual test, and not bleed over into other tests.

[bradhowes]

Hi @mbrandonw -- thanks for the quick response. I do setup an ImmediateClock when I create the TestStore. Now, I do not do this for the snapshot test -- I'm just creating a normal store via StoreOf. Again, if I disable the snapshot test everything works without any issue, but if I enable the snapshot test, then I get a failure and that strange Unimplemented: ContinuousClock.now message.

[mbrandonw]

I think my response is still relevant. Since snapshot tests must use a real store, there is a chance that after the snapshot, and after the test method finishes, some effect is still executing and causing the failure. That would cause it to bleed over into a test that does not have a problem, and makes it hard to track down what test is the culprit.

That is one theory I have for this. Another is that perhaps the entry point of the app is bleeding over into your tests.