PVC protection unit tests flake

[jsafrane]

What happened:
Random unit tests flake with Claim fake-pvc does not have clone protection finalizer set. The failure rate is very small, "4989 runs so far, 3 failures (0.06%)". And it's always a different test.
Example from our CI.

How to reproduce it:
With v5.2.0 tag:

$ # install `stress` tool
$ go install golang.org/x/tools/cmd/stress@latest

$ # Compile the unit tests
$ git checkout v5.2.0
$ cd pkg/controller
$ go test -c -race

$ stress -p 100 ./controller.test

Where 100 is nr. of tests to run in parallel. Using ~ 5x nr. of my CPUs (incl. hyperthreading ones), still got CPUs ~50% idle. I got 5 flakes in 30 minutes.

Individual failures:

--- FAIL: TestProvisionFromPVC (7.15s)
    --- FAIL: TestProvisionFromPVC/provision_with_pvc_data_source_different_storage_classes_when_CrossNamespaceVolumeDataSource_feature_enabled (0.11s)
        controller_test.go:6690: Claim fake-pvc does not have clone protection finalizer set

--- FAIL: TestProvisionFromPVC (7.15s)
    --- FAIL: TestProvisionFromPVC/provision_nil_mode_data_source_is_nil (0.11s)
        controller_test.go:6690: Claim fake-pvc does not have clone protection finalizer set

--- FAIL: TestProvisionFromPVC (7.17s)
    --- FAIL: TestProvisionFromPVC/provision_with_pvc_data_source_different_storage_classes_when_CrossNamespaceVolumeDataSource_feature_enabled (0.11s)
        controller_test.go:6690: Claim fake-pvc does not have clone protection finalizer set

--- FAIL: TestProvisionFromPVC (7.04s)
    --- FAIL: TestProvisionFromPVC/provision_with_pvc_data_source_destination_too_small (0.12s)
        controller_test.go:6690: Claim fake-pvc does not have clone protection finalizer set

--- FAIL: TestProvisionFromPVC (7.16s)
    --- FAIL: TestProvisionFromPVC/provision_block_data_source_is_block (0.12s)
        controller_test.go:6690: Claim block-pv does not have clone protection finalizer set

[jsafrane]

/kind flake
/help
/good-first-issue

[k8s-ci-robot]

@jsafrane:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/kind flake
/help
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[darshansreenivas]

/assign