Executor pods not explicitly deleted When SparkApplication enters INVALIDATING state

[Shekharrajak]

What happened?

• ✋ I have searched the open/closed issues and my issue is not listed.

When updating executor instances, SparkApplication correctly transitions to INVALIDATING state, but executor pods are not explicitly deleted by the operator. Cleanup relies on Spark's automatic cleanup when the driver terminates, creating a timing window where executor pods may remain running and potentially cause resource leaks or state inconsistencies.

Steps to Reproduce

1. Create SparkApplication with 1 executor

kubectl apply -f - <<EOF
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-test
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: docker.io/library/spark:4.0.0
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar
  arguments: ["10000"]
  sparkVersion: 4.0.0
  driver:
    cores: 1
    memory: 512m
    serviceAccount: spark-operator-spark
  executor:
    instances: 1
    cores: 1
    memory: 512m
EOF

2. Wait for RUNNING state

kubectl wait --for=jsonpath='{.status.applicationState.state}'=RUNNING sparkapplication/spark-pi-test --timeout=60s
kubectl get pods -l spark-app-name=spark-pi-test

3. Update executor instances

kubectl patch sparkapplication spark-pi-test --type='merge' -p='{"spec":{"executor":{"instances":3}}}'

4. Observe the issue

# Check application state (should be INVALIDATING)
kubectl get sparkapplication spark-pi-test -o jsonpath='{.status.applicationState.state}'

# Check if old executor pods are still running
kubectl get pods -l spark-role=executor | grep spark-pi

# Check operator logs - executor pods are NOT explicitly deleted
kubectl logs -n spark-operator -l app=spark-operator-controller | grep -i "deleting executor pod"

Expected Behavior

Executor pods should be explicitly deleted by the operator when application enters INVALIDATING state, ensuring deterministic cleanup.

Environment & Versions

• Kubernetes Version:
• Spark Operator Version:
• Apache Spark Version:

Additional context

No response

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[Shekharrajak]

Looks like delete executor pod is missing here https://github.com/kubeflow/spark-operator/blob/master/internal/controller/sparkapplication/controller.go#L1126

func (r *Reconciler) deleteSparkResources(ctx context.Context, app *v1beta2.SparkApplication) error {
	if err := r.deleteDriverPod(ctx, app); err != nil {
		return err
	}


	if err := r.deleteWebUIService(ctx, app); err != nil {
		return err
	}

	if err := r.deleteWebUIIngress(ctx, app); err != nil {
		return err
	}

	return nil
}

[BhumikaTechHub]

Based on the above flow, as the SparkApplication enters the INVALIDATING state (e.g. as a result of an update to the number of executors), the operator is seen to use the implicit cleanup when terminating drivers as opposed to explicitly reconciling executor pods. This leaves a gap between the completion of old executor pods and the departure of the driver, potentially causing temporary resource underutilization or inconsistency of state, particularly with high frequency updates or limited cluster conditions.

As mentioned, deleteSparkResources at the moment performs the cleanup of driver pod, Web UI service, and ingress but it does not specifically mention executor pods. When the executor pods are not invalidated deterministically, their lifecycle is indirectly tied to the timing of driver shutdown being scheduled instead of controller reconciliation and is somewhat inconsistent with the otherwise explicit resource management model of operator.

This difference is significant in those environments where applications need to be reconfigured regularly or operator restarts and retries are likely to occur because the use of only Spark-side cleanup may render behavior less predictable under Kubernetes controller behaviour.

It would be useful to make clear that such omission is deliberate (to not to interfere with the lifecycle management of Spark itself), or whether deletion of executor pod during INVALIDATING should be handled similarly to driver and UI resources. At the least, recording this behaviour and its consequences would allow users to have a more informed view of the existing cleanup semantics.

[Denyme24]

Hey @Shekharrajak @andreyvelich @nabuskey @ImpSy , I’d like to work on this issue.

Root Cause

When a SparkApplication moves to the INVALIDATING state (e.g., during spec updates), the operator deletes only the driver pod and relies on Spark’s internal cleanup. This creates a timing window where executor pods can remain running.

Fix

Update deleteSparkResources() to explicitly clean up executor pods during invalidation:

• Add deleteExecutorPods() to fetch and delete all executor pods
• Delete executor pods before the driver pod for deterministic cleanup
• Keep WebUI service and ingress cleanup unchanged

[Denyme24]

/assign