Add instructions on how to build dev environment (kubernetes#2472)

This commit enhances the CI playbooks to allow deploying from `master`
branch and adds documentation on how to use them to deploy a development
environment.

Author: dulek

docs/developers-guide.md

@@ -42,6 +42,65 @@ Choose the one you are familiar with and easy to customize. Config the cluster w
 
 Using kubeadm, openstack-cloud-controller-manager can be deployed easily with predefined manifests, see the [deployment guide with kubeadm](openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#deploy-a-kubernetes-cluster-with-openstack-cloud-controller-manager-using-kubeadm).
 
+### DevStack-based testing environment
+You can also use our CI scripts to setup a simple development environment based on DevStack and k3s. To do so you need a fresh VM with Ubuntu 22.04. We've tested this with 4 vCPUs and 16 GB of RAM and that's recommended, but we never tested the lower bound, so feel free to try with less resources.
+
+Once the VM is up make sure your SSH keys allow logging in as `ubuntu` user and from your PC and cloud-provider-openstack directory run:
+
+```
+ansible-playbook -v \
+  --user ubuntu \
+  --inventory <PUBLIC_IP_OF_YOUR_VM>, \
+  --ssh-common-args "-o StrictHostKeyChecking=no" \
+  tests/playbooks/test-occm-e2e.yaml \
+  -e octavia_provider=amphora \
+  -e run_e2e=false
+```
+
+After it finishes you should be able to access both DevStack and Kubernetes:
+
+```
+# SSH to the VM
+$ ssh ubuntu@<PUBLIC_IP_OF_YOUR_VM>
+
+# Apparently we install K8s in root
+$ sudo su
+
+# Load OpenStack credentials
+$ source /home/stack/devstack/openrc admin admin
+
+# List all pods in K8s
+$ kubectl get pods -A
+NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
+kube-system   openstack-cloud-controller-manager-55h4w   1/1     Running   0          56m
+kube-system   local-path-provisioner-5d56847996-5fqmn    1/1     Running   0          60m
+kube-system   coredns-5c6b6c5476-l5dz4                   1/1     Running   0          60m
+
+# Deploy a simple pod
+$ kubectl create deploy test --image quay.io/kuryr/demo:latest
+deployment.apps/test created
+
+# Expose it as a LoadBalancer Service
+$ kubectl expose deploy test --type LoadBalancer --target-port 8080 --port 80
+service/test exposed
+
+# Check if LB got created
+$ openstack loadbalancer list
++--------------------------------------+--------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
+| id                                   | name                                 | project_id                       | vip_address | provisioning_status | operating_status | provider |
++--------------------------------------+--------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
+| 9873d6d7-8ff1-4b5e-a8f0-6bb61f4dd58f | kube_service_kubernetes_default_test | deca4a226df049a689992105a65fbb66 | 10.1.0.36   | ACTIVE              | ONLINE           | amphora  |
++--------------------------------------+--------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
+
+# Get the external IP of the service
+$ kubectl get svc test
+NAME   TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
+test   LoadBalancer   10.43.71.235   172.24.5.121   80:30427/TCP   41m
+
+# Call the LB
+$ curl 172.24.5.121
+test-846c6ffb69-w52vq: HELLO! I AM ALIVE!!!
+```
 
 ## Contribution
 Now you should have a kubernetes cluster running in openstack and openstack-cloud-controller-manager is deployed in the cluster. Over time, you may find a bug or have some feature requirements, it's time for contribution!

tests/ci-csi-cinder-e2e.sh

@@ -32,7 +32,6 @@ cd "${REPO_ROOT}" || exit 1
 # PULL_NUMBER and PULL_BASE_REF are Prow job environment variables
 PR_NUMBER="${PULL_NUMBER:-}"
 [[ -z $PR_NUMBER ]] && echo "PR_NUMBER is required" && exit 1
-PR_BRANCH="${PULL_BASE_REF:-master}"
 CONFIG_ANSIBLE="${CONFIG_ANSIBLE:-"true"}"
 RESOURCE_TYPE="${RESOURCE_TYPE:-"gce-project"}"
 ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
@@ -106,8 +105,7 @@ ansible-playbook -v \
   --inventory ${PUBLIC_IP}, \
   --ssh-common-args "-o StrictHostKeyChecking=no" \
   tests/playbooks/test-csi-cinder-e2e.yaml \
-  -e github_pr=${PR_NUMBER} \
-  -e github_pr_branch=${PR_BRANCH}
+  -e github_pr=${PR_NUMBER}
 exit_code=$?
 
 # Fetch cinder-csi tests logs for debugging purpose

tests/ci-csi-manila-e2e.sh

@@ -32,7 +32,6 @@ cd "${REPO_ROOT}" || exit 1
 # PULL_NUMBER and PULL_BASE_REF are Prow job environment variables
 PR_NUMBER="${PULL_NUMBER:-}"
 [[ -z $PR_NUMBER ]] && echo "PR_NUMBER is required" && exit 1
-PR_BRANCH="${PULL_BASE_REF:-master}"
 CONFIG_ANSIBLE="${CONFIG_ANSIBLE:-"true"}"
 RESOURCE_TYPE="${RESOURCE_TYPE:-"gce-project"}"
 ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
@@ -106,8 +105,7 @@ ansible-playbook -v \
   --inventory ${PUBLIC_IP}, \
   --ssh-common-args "-o StrictHostKeyChecking=no" \
   tests/playbooks/test-csi-manila-e2e.yaml \
-  -e github_pr=${PR_NUMBER} \
-  -e github_pr_branch=${PR_BRANCH}
+  -e github_pr=${PR_NUMBER}
 exit_code=$?
 
 # Fetch manila-csi tests results

tests/ci-occm-e2e.sh

@@ -32,7 +32,6 @@ cd "${REPO_ROOT}" || exit 1
 # PULL_NUMBER and PULL_BASE_REF are Prow job environment variables
 PR_NUMBER="${PULL_NUMBER:-}"
 [[ -z $PR_NUMBER ]] && echo "PR_NUMBER is required" && exit 1
-PR_BRANCH="${PULL_BASE_REF:-master}"
 CONFIG_ANSIBLE="${CONFIG_ANSIBLE:-"true"}"
 RESOURCE_TYPE="${RESOURCE_TYPE:-"gce-project"}"
 ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
@@ -108,8 +107,8 @@ ansible-playbook -v \
   --ssh-common-args "-o StrictHostKeyChecking=no" \
   tests/playbooks/test-occm-e2e.yaml \
   -e github_pr=${PR_NUMBER} \
-  -e github_pr_branch=${PR_BRANCH} \
-  -e octavia_provider=${OCTAVIA_PROVIDER}
+  -e octavia_provider=${OCTAVIA_PROVIDER} \
+  -e run_e2e=true
 exit_code=$?
 
 # Fetch devstack logs for debugging purpose

tests/playbooks/roles/install-cpo-occm/defaults/main.yaml

@@ -1,5 +1,5 @@
 ---
-github_pr: 123
+github_pr: ""
 devstack_workdir: "{{ ansible_user_dir }}/devstack"
 
 # Used for uploading image to local registry.

tests/playbooks/roles/install-cpo-occm/tasks/main.yaml

@@ -7,18 +7,24 @@
       mkdir -p $GOPATH/src/k8s.io; cd $_
       git clone https://github.com/kubernetes/cloud-provider-openstack
       cd cloud-provider-openstack
-      git fetch origin +refs/pull/{{ github_pr }}/merge
-      git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
+      if [[ {{ github_pr }} != "" ]]; then
+        git fetch origin +refs/pull/{{ github_pr }}/merge
+        git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
+      fi
 
 - name: Build and upload openstack-cloud-controller-manager image
   shell:
     executable: /bin/bash
     cmd: |
       cd $GOPATH/src/k8s.io/cloud-provider-openstack
+      VERSION="latest"
+      if [[ {{ github_pr }} != "" ]]; then
+        VERSION=v0.0.{{ github_pr }}
+      fi
 
       make push-multiarch-image-openstack-cloud-controller-manager \
         ARCHS='amd64' \
-        VERSION=v0.0.{{ github_pr }} \
+        VERSION=${VERSION} \
         REGISTRY={{ image_registry_host }}
 
 - name: Prepare openstack-cloud-controller-manager config
@@ -70,9 +76,13 @@
   shell:
     executable: /bin/bash
     cmd: |
+      VERSION="latest"
+      if [[ {{ github_pr }} != "" ]]; then
+        VERSION=v0.0.{{ github_pr }}
+      fi
       cd $GOPATH/src/k8s.io/cloud-provider-openstack
       # replace image with built image
-      sed -i "s#registry.k8s.io/provider-os/openstack-cloud-controller-manager:v1.28.0#{{ remote_registry_host }}/openstack-cloud-controller-manager:v0.0.{{ github_pr }}#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
+      sed -i "s#registry.k8s.io/provider-os/openstack-cloud-controller-manager:v1.28.0#{{ remote_registry_host }}/openstack-cloud-controller-manager:${VERSION}#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
       sed -i "s#node-role.kubernetes.io/control-plane: \"\"#node-role.kubernetes.io/control-plane: \"true\"#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
       sed -i "s#--v=1#--v=5#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
       cat manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml

tests/playbooks/roles/install-csi-cinder/defaults/main.yaml

@@ -1,5 +1,5 @@
 ---
-github_pr: 123
+github_pr: ""
 devstack_workdir: "{{ ansible_user_dir }}/devstack"
 
 # Used for uploading image to local registry.

tests/playbooks/roles/install-csi-cinder/tasks/main.yaml

@@ -7,18 +7,24 @@
       mkdir -p $GOPATH/src/k8s.io; cd $_
       git clone https://github.com/kubernetes/cloud-provider-openstack
       cd cloud-provider-openstack
-      git fetch origin +refs/pull/{{ github_pr }}/merge
-      git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
+      if [[ {{ github_pr }} != "" ]]; then
+        git fetch origin +refs/pull/{{ github_pr }}/merge
+        git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
+      fi
 
 - name: Build and upload cinder-csi-plugin image
   shell:
     executable: /bin/bash
     cmd: |
       cd $GOPATH/src/k8s.io/cloud-provider-openstack
+      VERSION="latest"
+      if [[ {{ github_pr }} != "" ]]; then
+        VERSION=v0.0.{{ github_pr }}
+      fi
 
       make push-multiarch-image-cinder-csi-plugin \
         ARCHS='amd64' \
-        VERSION=v0.0.{{ github_pr }} \
+        VERSION=${VERSION} \
         REGISTRY={{ image_registry_host }}
 
 - name: Prepare cloud config
@@ -59,9 +65,13 @@
 
       sed -i "/cloud\.conf/c\  cloud.conf: $b64data" manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml
 
+      VERSION="latest"
+      if [[ {{ github_pr }} != "" ]]; then
+        VERSION=v0.0.{{ github_pr }}
+      fi
       # replace image with built image
-      sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:v0.0.{{ github_pr }}#" manifests/cinder-csi-plugin/cinder-csi-controllerplugin.yaml
-      sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:v0.0.{{ github_pr }}#" manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml
+      sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:${VERSION}#" manifests/cinder-csi-plugin/cinder-csi-controllerplugin.yaml
+      sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:${VERSION}#" manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml
 
       sed -i "s#--v=1#--v=5#" manifests/cinder-csi-plugin/cinder-csi-controllerplugin.yaml
       sed -i "s#--v=1#--v=5#" manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml

tests/playbooks/roles/install-csi-manila/defaults/main.yaml

@@ -1,5 +1,5 @@
 ---
-github_pr: 123
+github_pr: ""
 devstack_workdir: "{{ ansible_user_dir }}/devstack"
 
 # Used for uploading image to local registry.

tests/playbooks/roles/install-csi-manila/tasks/main.yaml

@@ -7,19 +7,25 @@
       mkdir -p {{ ansible_user_dir }}/src/k8s.io; cd $_
       git clone https://github.com/kubernetes/cloud-provider-openstack
       cd cloud-provider-openstack
-      git fetch origin +refs/pull/{{ github_pr }}/merge
-      git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
+      if [[ {{ github_pr }} != "" ]]; then
+        git fetch origin +refs/pull/{{ github_pr }}/merge
+        git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
+      fi
 
 - name: Build and upload manila-csi-plugin image
   shell:
     executable: /bin/bash
     cmd: |
       cd {{ ansible_user_dir }}/src/k8s.io/cloud-provider-openstack
+      VERSION="latest"
+      if [[ {{ github_pr }} != "" ]]; then
+        VERSION=v0.0.{{ github_pr }}
+      fi
 
       make push-multiarch-image-manila-csi-plugin \
         ARCHS='amd64' \
-        REGISTRY={{ image_registry_host }} \
-        VERSION=v0.0.{{ github_pr }}
+        VERSION=${VERSION} \
+        REGISTRY={{ image_registry_host }}
 
 - name: Prepare cloud config
   shell:
@@ -118,12 +124,16 @@
   shell:
     executable: /bin/bash
     cmd: |
+      VERSION="latest"
+      if [[ {{ github_pr }} != "" ]]; then
+        VERSION=v0.0.{{ github_pr }}
+      fi
       cd {{ ansible_user_dir }}/src/k8s.io/cloud-provider-openstack/charts/manila-csi-plugin
       cat <<EOF >> override-helm-values.yaml
       csimanila:
         image:
           repository: {{ remote_registry_host }}/manila-csi-plugin
-          tag: v0.0.{{ github_pr }}
+          tag: ${VERSION}
       shareProtocols:
         - protocolSelector: NFS
           fsGroupPolicy: None

tests/playbooks/test-occm-e2e.yaml

@@ -28,6 +28,6 @@
     - role: install-docker-registry
       cert_hosts: ' ["{{ ansible_default_ipv4.address }}"]'
     - role: install-cpo-occm
-      run_e2e: true
+      run_e2e: "{{ run_e2e }}"
       octavia_provider: "{{ octavia_provider }}"
       environment: "{{ global_env }}"