add ppc64le cluster to prow #7862

upodroid · 2025-03-11T10:12:41Z

/cc @Rajalakshmi-Girish @ameukam @BenTheElder

k8s-ci-robot · 2025-03-11T10:12:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~infra/gcp/terraform/OWNERS~~ [upodroid]
~~kubernetes/OWNERS~~ [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-infra-ci-robot · 2025-03-11T10:16:20Z

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow workspace: default

Show Output

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.gcb_bucket.google_storage_bucket.bucket has changed
~ resource "google_storage_bucket" "bucket" {
        id                          = "k8s-infra-prow-gcb"
        name                        = "k8s-infra-prow-gcb"
        # (16 unchanged attributes hidden)

      ~ lifecycle_rule {
          - condition {
              - age                        = 7 -> null
              - days_since_custom_time     = 0 -> null
              - days_since_noncurrent_time = 0 -> null
              - matches_prefix             = [] -> null
              - matches_storage_class      = [] -> null
              - matches_suffix             = [] -> null
              - no_age                     = false -> null
              - num_newer_versions         = 0 -> null
              - send_age_if_zero           = true -> null
              - with_state                 = "ANY" -> null
            }
          + condition {
              + age                                     = 7
              + days_since_custom_time                  = 0
              + days_since_noncurrent_time              = 0
              + matches_prefix                          = []
              + matches_storage_class                   = []
              + matches_suffix                          = []
              + no_age                                  = false
              + num_newer_versions                      = 0
              + send_age_if_zero                        = true
              + send_days_since_custom_time_if_zero     = false
              + send_days_since_noncurrent_time_if_zero = false
              + send_num_newer_versions_if_zero         = false
              + with_state                              = "ANY"
            }

            # (1 unchanged block hidden)
        }

        # (2 unchanged blocks hidden)
    }

  # module.prow.google_container_cluster.primary has changed
~ resource "google_container_cluster" "primary" {
        id                                       = "projects/k8s-infra-prow/locations/us-central1/clusters/prow"
      ~ master_version                           = "1.30.4-gke.1348000" -> "1.30.9-gke.1046000"
        name                                     = "prow"
      ~ node_version                             = "1.30.3-gke.1969001" -> "1.30.9-gke.1046000"
        # (29 unchanged attributes hidden)

      ~ cluster_autoscaling {
          + auto_provisioning_locations = []
            # (2 unchanged attributes hidden)
        }

      ~ node_pool {
            name                        = "prod-v1"
          ~ version                     = "1.30.3-gke.1969001" -> "1.30.9-gke.1046000"
            # (6 unchanged attributes hidden)

            # (5 unchanged blocks hidden)
        }

      ~ node_pool_defaults {
          ~ node_config_defaults {
              + insecure_kubelet_readonly_port_enabled = "TRUE"
                # (1 unchanged attribute hidden)

                # (1 unchanged block hidden)
            }
        }

      + secret_manager_config {
          + enabled = false
        }

        # (26 unchanged blocks hidden)
    }

  # module.prow_bucket.google_storage_bucket.bucket has changed
~ resource "google_storage_bucket" "bucket" {
        id                          = "kubernetes-ci-logs"
        name                        = "kubernetes-ci-logs"
        # (15 unchanged attributes hidden)

      + lifecycle_rule {
          + action {
              + type = "Delete"
            }
          + condition {
              + age                                     = 90
              + days_since_custom_time                  = 0
              + days_since_noncurrent_time              = 0
              + matches_prefix                          = []
              + matches_storage_class                   = []
              + matches_suffix                          = []
              + no_age                                  = false
              + num_newer_versions                      = 0
              + send_age_if_zero                        = true
              + send_days_since_custom_time_if_zero     = false
              + send_days_since_noncurrent_time_if_zero = false
              + send_num_newer_versions_if_zero         = false
              + with_state                              = "ANY"
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.testgrid_config_bucket.google_storage_bucket.bucket has changed
~ resource "google_storage_bucket" "bucket" {
        id                          = "k8s-testgrid-config"
        name                        = "k8s-testgrid-config"
        # (16 unchanged attributes hidden)

      ~ lifecycle_rule {
          - condition {
              - age                        = 90 -> null
              - days_since_custom_time     = 0 -> null
              - days_since_noncurrent_time = 0 -> null
              - matches_prefix             = [] -> null
              - matches_storage_class      = [] -> null
              - matches_suffix             = [] -> null
              - no_age                     = false -> null
              - num_newer_versions         = 0 -> null
              - send_age_if_zero           = true -> null
              - with_state                 = "ANY" -> null
            }
          + condition {
              + age                                     = 90
              + days_since_custom_time                  = 0
              + days_since_noncurrent_time              = 0
              + matches_prefix                          = []
              + matches_storage_class                   = []
              + matches_suffix                          = []
              + no_age                                  = false
              + num_newer_versions                      = 0
              + send_age_if_zero                        = true
              + send_days_since_custom_time_if_zero     = false
              + send_days_since_noncurrent_time_if_zero = false
              + send_num_newer_versions_if_zero         = false
              + with_state                              = "ANY"
            }

            # (1 unchanged block hidden)
        }

        # (2 unchanged blocks hidden)
    }

  # module.utility_cluster.google_container_cluster.primary has changed
~ resource "google_container_cluster" "primary" {
        id                                       = "projects/k8s-infra-prow/locations/us-central1/clusters/utility"
      ~ master_version                           = "1.30.3-gke.1639000" -> "1.30.9-gke.1127000"
        name                                     = "utility"
      ~ node_version                             = "1.29.6-gke.1326000" -> "1.30.9-gke.1127000"
        # (29 unchanged attributes hidden)

      ~ cluster_autoscaling {
          + auto_provisioning_locations = []
            # (2 unchanged attributes hidden)
        }

      ~ node_pool {
            name                        = "prod-v1"
          ~ version                     = "1.29.6-gke.1326000" -> "1.30.9-gke.1127000"
            # (6 unchanged attributes hidden)

            # (5 unchanged blocks hidden)
        }

      ~ node_pool_defaults {
          ~ node_config_defaults {
              + insecure_kubelet_readonly_port_enabled = "TRUE"
                # (1 unchanged attribute hidden)

                # (1 unchanged block hidden)
            }
        }

      + secret_manager_config {
          + enabled = false
        }

        # (26 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place

Terraform will perform the following actions:

  # google_iam_workload_identity_pool_provider.ppc64le will be updated in-place
~ resource "google_iam_workload_identity_pool_provider" "ppc64le" {
        id                                 = "projects/k8s-infra-prow/locations/global/workloadIdentityPools/ibm-clusters/providers/ppc64le"
        name                               = "projects/16065310909/locations/global/workloadIdentityPools/ibm-clusters/providers/ppc64le"
        # (6 unchanged attributes hidden)

      ~ oidc {
          ~ allowed_audiences = [
              + "sts.googleapis.com",
            ]
            # (2 unchanged attributes hidden)
        }
    }

  # module.prow_bucket.google_storage_bucket.bucket will be updated in-place
~ resource "google_storage_bucket" "bucket" {
        id                          = "kubernetes-ci-logs"
        name                        = "kubernetes-ci-logs"
        # (15 unchanged attributes hidden)

      - lifecycle_rule {
          - action {
              - type = "Delete" -> null
            }
          - condition {
              - age                                     = 90 -> null
              - days_since_custom_time                  = 0 -> null
              - days_since_noncurrent_time              = 0 -> null
              - matches_prefix                          = [] -> null
              - matches_storage_class                   = [] -> null
              - matches_suffix                          = [] -> null
              - no_age                                  = false -> null
              - num_newer_versions                      = 0 -> null
              - send_age_if_zero                        = true -> null
              - send_days_since_custom_time_if_zero     = false -> null
              - send_days_since_noncurrent_time_if_zero = false -> null
              - send_num_newer_versions_if_zero         = false -> null
              - with_state                              = "ANY" -> null
            }
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

▶️ To apply this plan, comment:

atlantis apply -d infra/gcp/terraform/k8s-infra-prow

🚮 To delete this plan and lock, click here

🔁 To plan this project again, comment:

atlantis plan -d infra/gcp/terraform/k8s-infra-prow

Note: Objects have changed outside of Terraform
Plan: 0 to add, 2 to change, 0 to destroy.

⏩ To apply all unapplied plans from this Pull Request, comment:
```
atlantis apply
```
🚮 To delete all plans and locks from this Pull Request, comment:
```
atlantis unlock
```

infra/gcp/terraform/k8s-infra-prow/iam.tf

ameukam · 2025-03-11T10:23:59Z

kubernetes/gke-utility/argocd/clusters.yaml

+      engineVersion: v2
+      data:
+        name: ibm-ppc64le
+        server: https://73725434-jp-osa.lb.appdomain.cloud:6443


I guess the cluster creation was handled differently since I don't see a HCL reflecting it ? So what ever happens to this cluster is not on SIG K8S Infra ?

There isn't an IBM managed cluster with this architecture, so it's a cluster built from VMs. I did ask for the TF code for the infra to be merged into this repository.

ameukam · 2025-03-11T10:25:21Z

kubernetes/ibm-ppc64le/prow/test-pods-poddisruptionbudget.yaml

+# It is still possible to delete the pods via a normal delete call. See https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#the-eviction-api
+
+apiVersion: policy/v1
+kind: PodDisruptionBudget


Do we still want to do this ? I see #7785 but I'm not sure.
cc @BenTheElder

k8s-infra-ci-robot · 2025-03-13T19:28:42Z

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow workspace: default

Show Output

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.gcb_bucket.google_storage_bucket.bucket has changed
~ resource "google_storage_bucket" "bucket" {
        id                          = "k8s-infra-prow-gcb"
        name                        = "k8s-infra-prow-gcb"
        # (16 unchanged attributes hidden)

      ~ lifecycle_rule {
          - condition {
              - age                        = 7 -> null
              - days_since_custom_time     = 0 -> null
              - days_since_noncurrent_time = 0 -> null
              - matches_prefix             = [] -> null
              - matches_storage_class      = [] -> null
              - matches_suffix             = [] -> null
              - no_age                     = false -> null
              - num_newer_versions         = 0 -> null
              - send_age_if_zero           = true -> null
              - with_state                 = "ANY" -> null
            }
          + condition {
              + age                                     = 7
              + days_since_custom_time                  = 0
              + days_since_noncurrent_time              = 0
              + matches_prefix                          = []
              + matches_storage_class                   = []
              + matches_suffix                          = []
              + no_age                                  = false
              + num_newer_versions                      = 0
              + send_age_if_zero                        = true
              + send_days_since_custom_time_if_zero     = false
              + send_days_since_noncurrent_time_if_zero = false
              + send_num_newer_versions_if_zero         = false
              + with_state                              = "ANY"
            }

            # (1 unchanged block hidden)
        }

        # (2 unchanged blocks hidden)
    }

  # module.prow.google_container_cluster.primary has changed
~ resource "google_container_cluster" "primary" {
        id                                       = "projects/k8s-infra-prow/locations/us-central1/clusters/prow"
      ~ master_version                           = "1.30.4-gke.1348000" -> "1.30.9-gke.1046000"
        name                                     = "prow"
      ~ node_version                             = "1.30.3-gke.1969001" -> "1.30.9-gke.1046000"
        # (29 unchanged attributes hidden)

      ~ cluster_autoscaling {
          + auto_provisioning_locations = []
            # (2 unchanged attributes hidden)
        }

      ~ node_pool {
            name                        = "prod-v1"
          ~ version                     = "1.30.3-gke.1969001" -> "1.30.9-gke.1046000"
            # (6 unchanged attributes hidden)

            # (5 unchanged blocks hidden)
        }

      ~ node_pool_defaults {
          ~ node_config_defaults {
              + insecure_kubelet_readonly_port_enabled = "TRUE"
                # (1 unchanged attribute hidden)

                # (1 unchanged block hidden)
            }
        }

      + secret_manager_config {
          + enabled = false
        }

        # (26 unchanged blocks hidden)
    }

  # module.prow_bucket.google_storage_bucket.bucket has changed
~ resource "google_storage_bucket" "bucket" {
        id                          = "kubernetes-ci-logs"
        name                        = "kubernetes-ci-logs"
        # (15 unchanged attributes hidden)

      + lifecycle_rule {
          + action {
              + type = "Delete"
            }
          + condition {
              + age                                     = 90
              + days_since_custom_time                  = 0
              + days_since_noncurrent_time              = 0
              + matches_prefix                          = []
              + matches_storage_class                   = []
              + matches_suffix                          = []
              + no_age                                  = false
              + num_newer_versions                      = 0
              + send_age_if_zero                        = true
              + send_days_since_custom_time_if_zero     = false
              + send_days_since_noncurrent_time_if_zero = false
              + send_num_newer_versions_if_zero         = false
              + with_state                              = "ANY"
            }
        }

        # (2 unchanged blocks hidden)
    }

  # module.testgrid_config_bucket.google_storage_bucket.bucket has changed
~ resource "google_storage_bucket" "bucket" {
        id                          = "k8s-testgrid-config"
        name                        = "k8s-testgrid-config"
        # (16 unchanged attributes hidden)

      ~ lifecycle_rule {
          - condition {
              - age                        = 90 -> null
              - days_since_custom_time     = 0 -> null
              - days_since_noncurrent_time = 0 -> null
              - matches_prefix             = [] -> null
              - matches_storage_class      = [] -> null
              - matches_suffix             = [] -> null
              - no_age                     = false -> null
              - num_newer_versions         = 0 -> null
              - send_age_if_zero           = true -> null
              - with_state                 = "ANY" -> null
            }
          + condition {
              + age                                     = 90
              + days_since_custom_time                  = 0
              + days_since_noncurrent_time              = 0
              + matches_prefix                          = []
              + matches_storage_class                   = []
              + matches_suffix                          = []
              + no_age                                  = false
              + num_newer_versions                      = 0
              + send_age_if_zero                        = true
              + send_days_since_custom_time_if_zero     = false
              + send_days_since_noncurrent_time_if_zero = false
              + send_num_newer_versions_if_zero         = false
              + with_state                              = "ANY"
            }

            # (1 unchanged block hidden)
        }

        # (2 unchanged blocks hidden)
    }

  # module.utility_cluster.google_container_cluster.primary has changed
~ resource "google_container_cluster" "primary" {
        id                                       = "projects/k8s-infra-prow/locations/us-central1/clusters/utility"
      ~ master_version                           = "1.30.3-gke.1639000" -> "1.30.9-gke.1127000"
        name                                     = "utility"
      ~ node_version                             = "1.29.6-gke.1326000" -> "1.30.9-gke.1127000"
        # (29 unchanged attributes hidden)

      ~ cluster_autoscaling {
          + auto_provisioning_locations = []
            # (2 unchanged attributes hidden)
        }

      ~ node_pool {
            name                        = "prod-v1"
          ~ version                     = "1.29.6-gke.1326000" -> "1.30.9-gke.1127000"
            # (6 unchanged attributes hidden)

            # (5 unchanged blocks hidden)
        }

      ~ node_pool_defaults {
          ~ node_config_defaults {
              + insecure_kubelet_readonly_port_enabled = "TRUE"
                # (1 unchanged attribute hidden)

                # (1 unchanged block hidden)
            }
        }

      + secret_manager_config {
          + enabled = false
        }

        # (26 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place

Terraform will perform the following actions:

  # google_iam_workload_identity_pool_provider.ppc64le will be updated in-place
~ resource "google_iam_workload_identity_pool_provider" "ppc64le" {
        id                                 = "projects/k8s-infra-prow/locations/global/workloadIdentityPools/ibm-clusters/providers/ppc64le"
        name                               = "projects/16065310909/locations/global/workloadIdentityPools/ibm-clusters/providers/ppc64le"
        # (6 unchanged attributes hidden)

      ~ oidc {
          ~ allowed_audiences = [
              + "sts.googleapis.com",
            ]
            # (2 unchanged attributes hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

▶️ To apply this plan, comment:

atlantis apply -d infra/gcp/terraform/k8s-infra-prow

🚮 To delete this plan and lock, click here

🔁 To plan this project again, comment:

atlantis plan -d infra/gcp/terraform/k8s-infra-prow

Note: Objects have changed outside of Terraform
Plan: 0 to add, 1 to change, 0 to destroy.

⏩ To apply all unapplied plans from this Pull Request, comment:
```
atlantis apply
```
🚮 To delete all plans and locks from this Pull Request, comment:
```
atlantis unlock
```

ameukam · 2025-03-14T16:56:05Z

/lgtm

infra/gcp/terraform/k8s-infra-prow/buckets.tf

ameukam · 2025-03-14T16:58:16Z

infra/gcp/terraform/k8s-infra-prow/iam.tf

@@ -138,3 +138,38 @@ resource "google_pubsub_topic_iam_binding" "read_binding" {
    "serviceAccount:[email protected]",
  ]
 }
+
+# https://cloud.google.com/iam/docs/workload-identity-federation-with-kubernetes#kubernetes
+# This is a community owned K8s Cluster inside IBM Cloud, have a look at the infra/ibm/terraform folder for more details


@Rajalakshmi-Girish I guess we have a follow-up PR to add the subpath defined ?

upodroid · 2025-03-15T15:03:06Z

/hold cancel

k8s-infra-ci-robot · 2025-03-15T15:07:53Z

Locks and plans deleted for the projects and workspaces modified in this pull request:

dir: infra/gcp/terraform/k8s-infra-prow workspace: default

add ppc64le cluster to prow

f307a12

k8s-ci-robot requested review from ameukam, BenTheElder and Rajalakshmi-Girish March 11, 2025 10:12

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Mar 11, 2025

ameukam reviewed Mar 11, 2025

View reviewed changes

infra/gcp/terraform/k8s-infra-prow/iam.tf Show resolved Hide resolved

ameukam reviewed Mar 11, 2025

View reviewed changes

reconcile infra diff

d95cf8b

k8s-ci-robot added area/infra Infrastructure management, infrastructure design, code in infra/ area/infra/gcp Issues or PRs related to Kubernetes GCP infrastructure area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters labels Mar 13, 2025

k8s-ci-robot assigned ameukam Mar 14, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 14, 2025

ameukam reviewed Mar 14, 2025

View reviewed changes

infra/gcp/terraform/k8s-infra-prow/buckets.tf Show resolved Hide resolved

ameukam reviewed Mar 14, 2025

View reviewed changes

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 15, 2025

k8s-ci-robot merged commit fc5a510 into kubernetes:main Mar 15, 2025
5 checks passed

k8s-ci-robot added this to the v1.33 milestone Mar 15, 2025

upodroid mentioned this pull request Mar 15, 2025

run our first ppc64le job kubernetes/test-infra#34503

Merged

This was referenced Mar 17, 2025

Umbrella issue: Onboarding ppc64le build cluster to k8s community's Prow #7888

Open

Add ClusterSecretStore for IBM provider's Secrets Manager #7904

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add ppc64le cluster to prow #7862

add ppc64le cluster to prow #7862

Uh oh!

upodroid commented Mar 11, 2025

Uh oh!

k8s-ci-robot commented Mar 11, 2025

Uh oh!

k8s-infra-ci-robot commented Mar 11, 2025

Uh oh!

Uh oh!

ameukam Mar 11, 2025

Uh oh!

upodroid Mar 11, 2025

Uh oh!

ameukam Mar 11, 2025

Uh oh!

k8s-infra-ci-robot commented Mar 13, 2025

Uh oh!

ameukam commented Mar 14, 2025

Uh oh!

Uh oh!

ameukam Mar 14, 2025

Uh oh!

upodroid commented Mar 15, 2025

Uh oh!

Uh oh!

k8s-infra-ci-robot commented Mar 15, 2025

Uh oh!

Uh oh!

add ppc64le cluster to prow #7862

add ppc64le cluster to prow #7862

Uh oh!

Conversation

upodroid commented Mar 11, 2025

Uh oh!

k8s-ci-robot commented Mar 11, 2025

Uh oh!

k8s-infra-ci-robot commented Mar 11, 2025

Uh oh!

Uh oh!

ameukam Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

upodroid Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

ameukam Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

k8s-infra-ci-robot commented Mar 13, 2025

Uh oh!

ameukam commented Mar 14, 2025

Uh oh!

Uh oh!

ameukam Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

upodroid commented Mar 15, 2025

Uh oh!

Uh oh!

k8s-infra-ci-robot commented Mar 15, 2025

Uh oh!

Uh oh!