Mount Point Leak in deploy/undeploy Cycles

[frobware]

Normal day-to-day usage of make deploy/undeploy cycles for bpfman-operator development causes exponential growth in mount entries within the cluster control plane, eventually rendering the cluster unusable.

Cycle 1 - Current mount count: 80
Cycle 2 - Current mount count: 82
Cycle 3 - Current mount count: 86
Cycle 4 - Current mount count: 95
Cycle 5 - Current mount count: 111
Cycle 6 - Current mount count: 143
Cycle 7 - Current mount count: 207
Cycle 8 - Current mount count: 335
Cycle 9 - Current mount count: 591
Cycle 10 - Current mount count: 1103
Cycle 11 - Current mount count: 2127
Cycle 12 - Current mount count: 4175
Cycle 13 - Current mount count: 8271
Cycle 14 - Current mount count: 16463
Cycle 15 - Current mount count: 32847

The growth follows an exponential pattern, approximately doubling each cycle after cycle 7. This indicates mount points are not being properly cleaned up during undeploy operations.

Impact

• Development Workflow Disruption: Clusters become unusable after routine deploy/undeploy cycles
• System Instability: High mount counts can cause kernel resource exhaustion
• Deployment Failures: Eventually pods fail to enter running state within timeout periods
• Performance Degradation: Mount table operations become increasingly slow

Reproduction Steps

1. Create a Kind cluster: make setup-kind
2. Monitor mount points in real-time (optional):

   while :; do
   echo -n "$(date): "; docker exec bpfman-deployment-control-plane wc -l /proc/mounts
   sleep 5
   done

3. Perform normal development cycles:
   • make deploy
   • make undeploy
   • Repeat
4. Observe mount count growth with each cycle
5. After ~15 cycles, deployments timeout waiting for pods to become ready

Environment

• Platform: KIND cluster on Linux (or OpenShift)
• Cluster: bpfman-deployment (default KIND cluster name)
• Namespace: bpfman
• Development Cycles: 15 deploy/undeploy cycles over ~9 minutes

Note: This issue is also reproducible on OpenShift clusters, where remediation requires rebooting affected nodes.

Expected Behaviour

Mount entries should return to baseline levels after each undeploy operation, with minimal growth over time.

Actual Behaviour

Mount entries accumulate exponentially, never being fully cleaned up during undeploy operations, eventually breaking the development cluster.

This issue significantly impacts the development workflow as clusters become unusable after routine development cycles. After ~15 deploy/undeploy cycles, the cluster reaches system limits (32,767 mount entries) and deployments begin failing.

Production Impact: On OpenShift clusters, this issue manifests similarly and requires node reboots to remediate, making it a critical issue for production deployments.

The issue likely stems from incomplete cleanup during the undeploy process, where mount points created during deployment are not properly removed.

Suggested Investigation

An initial avenue of investigation would be the CSI driver, as it manages volume mounts and may not be properly cleaning up mount points during undeploy operations. The exponential growth pattern suggests mount points are being created but never removed.

[frobware]

Reproducer script:

#!/usr/bin/env bash

set -euo pipefail

# Stress test for make deploy/undeploy cycle with mount monitoring
kind_cluster_name=${KIND_CLUSTER_NAME:-bpfman-deployment}
namespace=${NAMESPACE:-bpfman}
max_mounts=${MAX_MOUNTS:-32767}

echo "Starting deploy/undeploy stress test"
echo "KIND cluster: $kind_cluster_name"
echo "Target namespace: $namespace"
echo "Max mounts threshold: $max_mounts"
echo

wait_for_pods_running() {
    local namespace=$1
    local timeout=300  # 5 minutes timeout
    local start_time
    start_time=$(date +%s)

    echo "Waiting for all pods in namespace $namespace to be running..."

    while true; do
        local current_time
        current_time=$(date +%s)
        local elapsed=$((current_time - start_time))

        if [ $elapsed -ge $timeout ]; then
            echo "Timeout waiting for pods to be running after ${timeout}s"
            return 1
        fi

        local not_running_pods
        not_running_pods=$(kubectl get pods -n "$namespace" --no-headers 2>/dev/null | awk '$3 != "Running" {print $1}' | wc -l)
        local total_pods
        total_pods=$(kubectl get pods -n "$namespace" --no-headers 2>/dev/null | wc -l)

        if [ "$total_pods" -eq 0 ]; then
            echo "No pods found in namespace $namespace, waiting..."
            sleep 5
            continue
        fi

        if [ "$not_running_pods" -eq 0 ]; then
            echo "All $total_pods pod(s) in namespace $namespace are running"
            return 0
        fi

        echo "Waiting for $not_running_pods of $total_pods pod(s) to be running... (${elapsed}s elapsed)"
        sleep 5
    done
}

get_mount_count() {
    local count
    count=$(docker exec "$kind_cluster_name-control-plane" wc -l /proc/mounts 2>/dev/null | awk '{print $1}')
    echo "$count"
}

run_deploy() {
    echo "Running make deploy..."
    if ! make deploy; then
        echo "Deploy failed!"
        return 1
    fi
}

run_undeploy() {
    echo "Running make undeploy..."
    if ! make undeploy; then
        echo "Undeploy failed!"
        return 1
    fi
}

cycle=0
while true; do
    cycle=$((cycle + 1))
    echo "=== Cycle $cycle ==="

    if ! run_deploy; then
        echo "Deploy failed on cycle $cycle"
        exit 1
    fi

    if ! wait_for_pods_running "$namespace"; then
        echo "Pods failed to become running on cycle $cycle"
        exit 1
    fi

    mount_count=$(get_mount_count)
    echo "Cycle $cycle - Current mount count: $mount_count"

    if [ "$mount_count" -ge "$max_mounts" ]; then
        echo "Mount count threshold reached: $mount_count >= $max_mounts"
        echo "Stopping stress test after $cycle cycles"
        break
    fi

    if ! run_undeploy; then
        echo "Undeploy failed on cycle $cycle"
        exit 1
    fi

    echo "Cycle $cycle completed successfully"
    echo
done

echo "Stress test completed after $cycle cycles"
echo "Cycle $cycle - Final mount count: $mount_count"