Skip to content

Commit 96f2edb

Browse files
authored
Add instructions on how to build dev environment (kubernetes#2472)
This commit enhances the CI playbooks to allow deploying from `master` branch and adds documentation on how to use them to deploy a development environment.
1 parent 9e7794b commit 96f2edb

File tree

11 files changed

+111
-27
lines changed

11 files changed

+111
-27
lines changed

docs/developers-guide.md

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,65 @@ Choose the one you are familiar with and easy to customize. Config the cluster w
4242

4343
Using kubeadm, openstack-cloud-controller-manager can be deployed easily with predefined manifests, see the [deployment guide with kubeadm](openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#deploy-a-kubernetes-cluster-with-openstack-cloud-controller-manager-using-kubeadm).
4444

45+
### DevStack-based testing environment
46+
You can also use our CI scripts to setup a simple development environment based on DevStack and k3s. To do so you need a fresh VM with Ubuntu 22.04. We've tested this with 4 vCPUs and 16 GB of RAM and that's recommended, but we never tested the lower bound, so feel free to try with less resources.
47+
48+
Once the VM is up make sure your SSH keys allow logging in as `ubuntu` user and from your PC and cloud-provider-openstack directory run:
49+
50+
```
51+
ansible-playbook -v \
52+
--user ubuntu \
53+
--inventory <PUBLIC_IP_OF_YOUR_VM>, \
54+
--ssh-common-args "-o StrictHostKeyChecking=no" \
55+
tests/playbooks/test-occm-e2e.yaml \
56+
-e octavia_provider=amphora \
57+
-e run_e2e=false
58+
```
59+
60+
After it finishes you should be able to access both DevStack and Kubernetes:
61+
62+
```
63+
# SSH to the VM
64+
$ ssh ubuntu@<PUBLIC_IP_OF_YOUR_VM>
65+
66+
# Apparently we install K8s in root
67+
$ sudo su
68+
69+
# Load OpenStack credentials
70+
$ source /home/stack/devstack/openrc admin admin
71+
72+
# List all pods in K8s
73+
$ kubectl get pods -A
74+
NAMESPACE NAME READY STATUS RESTARTS AGE
75+
kube-system openstack-cloud-controller-manager-55h4w 1/1 Running 0 56m
76+
kube-system local-path-provisioner-5d56847996-5fqmn 1/1 Running 0 60m
77+
kube-system coredns-5c6b6c5476-l5dz4 1/1 Running 0 60m
78+
79+
# Deploy a simple pod
80+
$ kubectl create deploy test --image quay.io/kuryr/demo:latest
81+
deployment.apps/test created
82+
83+
# Expose it as a LoadBalancer Service
84+
$ kubectl expose deploy test --type LoadBalancer --target-port 8080 --port 80
85+
service/test exposed
86+
87+
# Check if LB got created
88+
$ openstack loadbalancer list
89+
+--------------------------------------+--------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
90+
| id | name | project_id | vip_address | provisioning_status | operating_status | provider |
91+
+--------------------------------------+--------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
92+
| 9873d6d7-8ff1-4b5e-a8f0-6bb61f4dd58f | kube_service_kubernetes_default_test | deca4a226df049a689992105a65fbb66 | 10.1.0.36 | ACTIVE | ONLINE | amphora |
93+
+--------------------------------------+--------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
94+
95+
# Get the external IP of the service
96+
$ kubectl get svc test
97+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
98+
test LoadBalancer 10.43.71.235 172.24.5.121 80:30427/TCP 41m
99+
100+
# Call the LB
101+
$ curl 172.24.5.121
102+
test-846c6ffb69-w52vq: HELLO! I AM ALIVE!!!
103+
```
45104

46105
## Contribution
47106
Now you should have a kubernetes cluster running in openstack and openstack-cloud-controller-manager is deployed in the cluster. Over time, you may find a bug or have some feature requirements, it's time for contribution!

tests/ci-csi-cinder-e2e.sh

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,6 @@ cd "${REPO_ROOT}" || exit 1
3232
# PULL_NUMBER and PULL_BASE_REF are Prow job environment variables
3333
PR_NUMBER="${PULL_NUMBER:-}"
3434
[[ -z $PR_NUMBER ]] && echo "PR_NUMBER is required" && exit 1
35-
PR_BRANCH="${PULL_BASE_REF:-master}"
3635
CONFIG_ANSIBLE="${CONFIG_ANSIBLE:-"true"}"
3736
RESOURCE_TYPE="${RESOURCE_TYPE:-"gce-project"}"
3837
ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
@@ -106,8 +105,7 @@ ansible-playbook -v \
106105
--inventory ${PUBLIC_IP}, \
107106
--ssh-common-args "-o StrictHostKeyChecking=no" \
108107
tests/playbooks/test-csi-cinder-e2e.yaml \
109-
-e github_pr=${PR_NUMBER} \
110-
-e github_pr_branch=${PR_BRANCH}
108+
-e github_pr=${PR_NUMBER}
111109
exit_code=$?
112110

113111
# Fetch cinder-csi tests logs for debugging purpose

tests/ci-csi-manila-e2e.sh

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,6 @@ cd "${REPO_ROOT}" || exit 1
3232
# PULL_NUMBER and PULL_BASE_REF are Prow job environment variables
3333
PR_NUMBER="${PULL_NUMBER:-}"
3434
[[ -z $PR_NUMBER ]] && echo "PR_NUMBER is required" && exit 1
35-
PR_BRANCH="${PULL_BASE_REF:-master}"
3635
CONFIG_ANSIBLE="${CONFIG_ANSIBLE:-"true"}"
3736
RESOURCE_TYPE="${RESOURCE_TYPE:-"gce-project"}"
3837
ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
@@ -106,8 +105,7 @@ ansible-playbook -v \
106105
--inventory ${PUBLIC_IP}, \
107106
--ssh-common-args "-o StrictHostKeyChecking=no" \
108107
tests/playbooks/test-csi-manila-e2e.yaml \
109-
-e github_pr=${PR_NUMBER} \
110-
-e github_pr_branch=${PR_BRANCH}
108+
-e github_pr=${PR_NUMBER}
111109
exit_code=$?
112110

113111
# Fetch manila-csi tests results

tests/ci-occm-e2e.sh

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,6 @@ cd "${REPO_ROOT}" || exit 1
3232
# PULL_NUMBER and PULL_BASE_REF are Prow job environment variables
3333
PR_NUMBER="${PULL_NUMBER:-}"
3434
[[ -z $PR_NUMBER ]] && echo "PR_NUMBER is required" && exit 1
35-
PR_BRANCH="${PULL_BASE_REF:-master}"
3635
CONFIG_ANSIBLE="${CONFIG_ANSIBLE:-"true"}"
3736
RESOURCE_TYPE="${RESOURCE_TYPE:-"gce-project"}"
3837
ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
@@ -108,8 +107,8 @@ ansible-playbook -v \
108107
--ssh-common-args "-o StrictHostKeyChecking=no" \
109108
tests/playbooks/test-occm-e2e.yaml \
110109
-e github_pr=${PR_NUMBER} \
111-
-e github_pr_branch=${PR_BRANCH} \
112-
-e octavia_provider=${OCTAVIA_PROVIDER}
110+
-e octavia_provider=${OCTAVIA_PROVIDER} \
111+
-e run_e2e=true
113112
exit_code=$?
114113

115114
# Fetch devstack logs for debugging purpose

tests/playbooks/roles/install-cpo-occm/defaults/main.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
github_pr: 123
2+
github_pr: ""
33
devstack_workdir: "{{ ansible_user_dir }}/devstack"
44

55
# Used for uploading image to local registry.

tests/playbooks/roles/install-cpo-occm/tasks/main.yaml

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,18 +7,24 @@
77
mkdir -p $GOPATH/src/k8s.io; cd $_
88
git clone https://github.com/kubernetes/cloud-provider-openstack
99
cd cloud-provider-openstack
10-
git fetch origin +refs/pull/{{ github_pr }}/merge
11-
git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
10+
if [[ {{ github_pr }} != "" ]]; then
11+
git fetch origin +refs/pull/{{ github_pr }}/merge
12+
git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
13+
fi
1214
1315
- name: Build and upload openstack-cloud-controller-manager image
1416
shell:
1517
executable: /bin/bash
1618
cmd: |
1719
cd $GOPATH/src/k8s.io/cloud-provider-openstack
20+
VERSION="latest"
21+
if [[ {{ github_pr }} != "" ]]; then
22+
VERSION=v0.0.{{ github_pr }}
23+
fi
1824
1925
make push-multiarch-image-openstack-cloud-controller-manager \
2026
ARCHS='amd64' \
21-
VERSION=v0.0.{{ github_pr }} \
27+
VERSION=${VERSION} \
2228
REGISTRY={{ image_registry_host }}
2329
2430
- name: Prepare openstack-cloud-controller-manager config
@@ -70,9 +76,13 @@
7076
shell:
7177
executable: /bin/bash
7278
cmd: |
79+
VERSION="latest"
80+
if [[ {{ github_pr }} != "" ]]; then
81+
VERSION=v0.0.{{ github_pr }}
82+
fi
7383
cd $GOPATH/src/k8s.io/cloud-provider-openstack
7484
# replace image with built image
75-
sed -i "s#registry.k8s.io/provider-os/openstack-cloud-controller-manager:v1.28.0#{{ remote_registry_host }}/openstack-cloud-controller-manager:v0.0.{{ github_pr }}#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
85+
sed -i "s#registry.k8s.io/provider-os/openstack-cloud-controller-manager:v1.28.0#{{ remote_registry_host }}/openstack-cloud-controller-manager:${VERSION}#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
7686
sed -i "s#node-role.kubernetes.io/control-plane: \"\"#node-role.kubernetes.io/control-plane: \"true\"#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
7787
sed -i "s#--v=1#--v=5#" manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
7888
cat manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml

tests/playbooks/roles/install-csi-cinder/defaults/main.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
github_pr: 123
2+
github_pr: ""
33
devstack_workdir: "{{ ansible_user_dir }}/devstack"
44

55
# Used for uploading image to local registry.

tests/playbooks/roles/install-csi-cinder/tasks/main.yaml

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,18 +7,24 @@
77
mkdir -p $GOPATH/src/k8s.io; cd $_
88
git clone https://github.com/kubernetes/cloud-provider-openstack
99
cd cloud-provider-openstack
10-
git fetch origin +refs/pull/{{ github_pr }}/merge
11-
git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
10+
if [[ {{ github_pr }} != "" ]]; then
11+
git fetch origin +refs/pull/{{ github_pr }}/merge
12+
git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
13+
fi
1214
1315
- name: Build and upload cinder-csi-plugin image
1416
shell:
1517
executable: /bin/bash
1618
cmd: |
1719
cd $GOPATH/src/k8s.io/cloud-provider-openstack
20+
VERSION="latest"
21+
if [[ {{ github_pr }} != "" ]]; then
22+
VERSION=v0.0.{{ github_pr }}
23+
fi
1824
1925
make push-multiarch-image-cinder-csi-plugin \
2026
ARCHS='amd64' \
21-
VERSION=v0.0.{{ github_pr }} \
27+
VERSION=${VERSION} \
2228
REGISTRY={{ image_registry_host }}
2329
2430
- name: Prepare cloud config
@@ -59,9 +65,13 @@
5965
6066
sed -i "/cloud\.conf/c\ cloud.conf: $b64data" manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml
6167
68+
VERSION="latest"
69+
if [[ {{ github_pr }} != "" ]]; then
70+
VERSION=v0.0.{{ github_pr }}
71+
fi
6272
# replace image with built image
63-
sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:v0.0.{{ github_pr }}#" manifests/cinder-csi-plugin/cinder-csi-controllerplugin.yaml
64-
sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:v0.0.{{ github_pr }}#" manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml
73+
sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:${VERSION}#" manifests/cinder-csi-plugin/cinder-csi-controllerplugin.yaml
74+
sed -i "s#registry.k8s.io/provider-os/cinder-csi-plugin:v1.28.0#{{ remote_registry_host }}/cinder-csi-plugin:${VERSION}#" manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml
6575
6676
sed -i "s#--v=1#--v=5#" manifests/cinder-csi-plugin/cinder-csi-controllerplugin.yaml
6777
sed -i "s#--v=1#--v=5#" manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml

tests/playbooks/roles/install-csi-manila/defaults/main.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
github_pr: 123
2+
github_pr: ""
33
devstack_workdir: "{{ ansible_user_dir }}/devstack"
44

55
# Used for uploading image to local registry.

tests/playbooks/roles/install-csi-manila/tasks/main.yaml

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,19 +7,25 @@
77
mkdir -p {{ ansible_user_dir }}/src/k8s.io; cd $_
88
git clone https://github.com/kubernetes/cloud-provider-openstack
99
cd cloud-provider-openstack
10-
git fetch origin +refs/pull/{{ github_pr }}/merge
11-
git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
10+
if [[ {{ github_pr }} != "" ]]; then
11+
git fetch origin +refs/pull/{{ github_pr }}/merge
12+
git checkout FETCH_HEAD; git checkout -b PR{{ github_pr }}
13+
fi
1214
1315
- name: Build and upload manila-csi-plugin image
1416
shell:
1517
executable: /bin/bash
1618
cmd: |
1719
cd {{ ansible_user_dir }}/src/k8s.io/cloud-provider-openstack
20+
VERSION="latest"
21+
if [[ {{ github_pr }} != "" ]]; then
22+
VERSION=v0.0.{{ github_pr }}
23+
fi
1824
1925
make push-multiarch-image-manila-csi-plugin \
2026
ARCHS='amd64' \
21-
REGISTRY={{ image_registry_host }} \
22-
VERSION=v0.0.{{ github_pr }}
27+
VERSION=${VERSION} \
28+
REGISTRY={{ image_registry_host }}
2329
2430
- name: Prepare cloud config
2531
shell:
@@ -118,12 +124,16 @@
118124
shell:
119125
executable: /bin/bash
120126
cmd: |
127+
VERSION="latest"
128+
if [[ {{ github_pr }} != "" ]]; then
129+
VERSION=v0.0.{{ github_pr }}
130+
fi
121131
cd {{ ansible_user_dir }}/src/k8s.io/cloud-provider-openstack/charts/manila-csi-plugin
122132
cat <<EOF >> override-helm-values.yaml
123133
csimanila:
124134
image:
125135
repository: {{ remote_registry_host }}/manila-csi-plugin
126-
tag: v0.0.{{ github_pr }}
136+
tag: ${VERSION}
127137
shareProtocols:
128138
- protocolSelector: NFS
129139
fsGroupPolicy: None

tests/playbooks/test-occm-e2e.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,6 @@
2828
- role: install-docker-registry
2929
cert_hosts: ' ["{{ ansible_default_ipv4.address }}"]'
3030
- role: install-cpo-occm
31-
run_e2e: true
31+
run_e2e: "{{ run_e2e }}"
3232
octavia_provider: "{{ octavia_provider }}"
3333
environment: "{{ global_env }}"

0 commit comments

Comments
 (0)