Add a test util function for killing the head Pod and wait.

[rueian]

Follow up for #3888 (comment).

Add a test util function for killing the head Pod and wait to replace the following in the code base.

// Delete the head Pod
err = test.Client().Core().CoreV1().Pods(namespace.Name).Delete(test.Ctx(), headPod.Name, metav1.DeleteOptions{})
g.Expect(err).NotTo(HaveOccurred())

PodUID := func(p *corev1.Pod) string { return string(p.UID) }
g.Eventually(HeadPod(test, rayCluster), TestTimeoutMedium).ShouldNot(WithTransform(PodUID, Equal(string(headPod.UID)))) // Use UID to check if the new head pod is created.
g.Eventually(HeadPod(test, rayCluster), TestTimeoutMedium).Should(WithTransform(func(p *corev1.Pod) string { return string(p.Status.Phase) }, Equal("Running")))

headPod, err = GetHeadPod(test, rayCluster) // Replace the old head pod
g.Expect(err).NotTo(HaveOccurred())

[owenowenisme]

Hi, may I take this issue, thanks!