Initial commit

Author: prabdeb

.github/ISSUE_TEMPLATE.md

@@ -21,7 +21,7 @@ IF SUFFICIENT INFORMATION IS NOT PROVIDED VIA THE FOLLOWING TEMPLATE THE ISSUE M
 ### Expected/desired behavior
 >
 
-### OS and Version?
+### Pipeline Agent?
 > Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
 
 ### Versions

.gitignore

@@ -20,8 +20,6 @@ parts/
 sdist/
 var/
 wheels/
-pip-wheel-metadata/
-share/python-wheels/
 *.egg-info/
 .installed.cfg
 *.egg
@@ -40,14 +38,12 @@ pip-delete-this-directory.txt
 # Unit test / coverage reports
 htmlcov/
 .tox/
-.nox/
 .coverage
 .coverage.*
 .cache
 nosetests.xml
 coverage.xml
 *.cover
-*.py,cover
 .hypothesis/
 .pytest_cache/
 
@@ -59,7 +55,6 @@ coverage.xml
 *.log
 local_settings.py
 db.sqlite3
-db.sqlite3-journal
 
 # Flask stuff:
 instance/
@@ -77,26 +72,11 @@ target/
 # Jupyter Notebook
 .ipynb_checkpoints
 
-# IPython
-profile_default/
-ipython_config.py
-
 # pyenv
 .python-version
 
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow
-__pypackages__/
-
-# Celery stuff
+# celery beat schedule file
 celerybeat-schedule
-celerybeat.pid
 
 # SageMath parsed files
 *.sage.py
@@ -122,8 +102,9 @@ venv.bak/
 
 # mypy
 .mypy_cache/
-.dmypy.json
-dmypy.json
 
-# Pyre type checker
-.pyre/
+# Framework related
+kubectl
+kind
+helm*
+linux-amd64/*

CHANGELOG.md

@@ -1,13 +1,19 @@
-## [project-title] Changelog
+## Automated Test Environment for AKS Applications Changelog
 
-<a name="x.y.z"></a>
-# x.y.z (yyyy-mm-dd)
+<a name="1.0.0"></a>
+# 1.0.0 (2020-08-03)
 
 *Features*
-* ...
+* Create/Delete `kind` cluster in CI environment
+* Optional - [Azure Key Vault Provider for Secrets Store CSI Driver](https://github.com/Azure/secrets-store-csi-driver-provider-azure) [installation](https://github.com/Azure/secrets-store-csi-driver-provider-azure#install-the-secrets-store-csi-driver-and-the-azure-keyvault-provider) and [configuration](https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/docs/service-principal-mode.md)
+* Optional - [Azure Container Registry (ACR) Image Pull Secret](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-kubernetes#create-an-image-pull-secret)
+* Install `helm` charts of application (values needed for integration test environment can be overwritten easily)
+* Validate if the respective Kubernetes `pods` are up and running (multiple `pods` can be provided that are having selector - `app.kubernetes.io/name`)
+* [Port-Froward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/#forward-a-local-port-to-a-port-on-the-pod) the respective Kubernetes `services` needed to perform integration testing (multiple `services` can be provided and the respective local ports will be starting from `8080` to `808[number of services]` maintaining the order as provided)
+
 
 *Bug Fixes*
-* ...
+* N/A
 
 *Breaking Changes*
-* ...
+* N/A

README.md

@@ -1,57 +1,77 @@
-# Project Name
+# Automated Test Environment for AKS Applications
 
-(short, 1-3 sentenced, description of the project)
+Framework to create Automated Test Environment using [kind](https://kind.sigs.k8s.io/) for testing Azure Kubernetes Service (AKS) based applications in CI (Continuous Integration) Pipeline (Azure DevOps), where required dependencies/infrastructure will be provisioned for executing automated tests and deprovisioned after completion.
+
+## Tools Used
+1. [kind](https://kind.sigs.k8s.io/) - a tool for running local Kubernetes clusters using Docker container “nodes”.
+2. [helm](https://helm.sh/) - the package manager for Kubernetes
+3. [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) - command line tool to control Kubernetes clusters
+4. [bash](https://www.gnu.org/software/bash/) - the GNU Project's shell
 
 ## Features
 
-This project framework provides the following features:
+This is a very simple Framework, that creates Automated Test Environment to enable automated testing of applications hosted in [Azure Kubernetes Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/) with the following features -
 
-* Feature 1
-* Feature 2
-* ...
+1. Create/Delete `kind` cluster in CI environment
+2. Optional - [Azure Key Vault Provider for Secrets Store CSI Driver](https://github.com/Azure/secrets-store-csi-driver-provider-azure) [installation](https://github.com/Azure/secrets-store-csi-driver-provider-azure#install-the-secrets-store-csi-driver-and-the-azure-keyvault-provider) and [configuration](https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/docs/service-principal-mode.md)
+3. Optional - [Azure Container Registry (ACR) Image Pull Secret](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-kubernetes#create-an-image-pull-secret)
+4. Install `helm` charts of application (values needed for integration test environment can be overwritten easily)
+5. Validate if the respective Kubernetes `pods` are up and running (multiple `pods` can be provided that are having selector - `app.kubernetes.io/name`)
+6. [Port-Froward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/#forward-a-local-port-to-a-port-on-the-pod) the respective Kubernetes `services` needed to perform integration testing (multiple `services` can be provided and the respective local ports will be starting from `8080` to `808[number of services]` maintaining the order as provided)
 
 ## Getting Started
 
-### Prerequisites
-
-(ideally very short, if any)
-
-- OS
-- Library version
-- ...
-
-### Installation
-
-(ideally very short)
-
-- npm install [package name]
-- mvn install
-- ...
-
-### Quickstart
-(Add steps to get up and running quickly)
-
-1. git clone [repository clone url]
-2. cd [respository name]
-3. ...
-
+This framework contains the following script and their options -
+
+1. `start.sh`: Download all the dependencies and create `kind` cluster -
+```sh
+# Usage: bash -f ./start.sh
+#        Supported Options -
+#               --kind-cluster-name=<Kind Cluster Name> (default INTEGRATION_TEST_CLUSTER)
+#               --kind-version (default v0.7.0)
+#               --kubectl-version (default v1.18.0)
+#               --helm-version (default v3.2.0)
+```
+2. `deploy.sh`: Deploy/Port-Forward application `helm chart` and enables optional features -
+```sh
+# Usage: bash -f ./deploy.sh
+#        Supported Options - 
+#              --csi-driver-enabled=<yes/no> (default no, if yes provide following two parameters)
+#              --csi-driver-sp-client-id=<Azure Service Principle ID, having access to Azure Key Vault>
+#              --csi-driver-sp-client-secret=<Azure Service Principle Secret, having access to Azure Key Vault>
+#              --acr-imagepullsecret-enabled=<yes/no> (default no, if yes provide following three parameters)
+#              --acr-imagepullsecret-sp-client-id=<Azure Service Principle ID, having access to Azure Container Registry>
+#              --acr-imagepullsecret-sp-client-secret=<Azure Service Principle Secret, having access to Azure Container Registry>
+#              --acr-full-name=<Azure Container Registry full name ex. example.azurecr.io>
+#              --helm-chart-path=<Helm Chart Folder Path or URL to .tgz file for the applications >
+#              --helm-chart-release-name=<Helm Release Name>
+#              --helm-chart-set-parameters=<","(comma) seprated Helm Set parameters needed to be overwritten for integration test env>
+#              --kubectl-check-services=<","(comma) seprated Pod names needed to be check if up and running>
+#              --kubectl-check-services-selector-label=<ex. app.kubernetes.io/name or name etc.> (default app.kubernetes.io/name)
+#              --kubectl-port-forward-services=<","(comma) seprated Service names needed to port-forward for testing>
+```
+3. `stop.sh`: Delete `kind` cluster -
+```sh
+# Usage: bash -f ./stop.sh
+#        Supported Options -
+#               --kind-cluster-name=<Kind Cluster Name> (default INTEGRATION_TEST_CLUSTER)
+```
 
 ## Demo
 
-A demo app is included to show how to use the project.
+For [demonstrating the framework](./sample) following setup has been used -
 
-To run the demo, follow these steps:
+* Sample Application [azure-vote](https://github.com/Azure-Samples/helm-charts/tree/master/chart-source/azure-vote)
+* [Azure DevOps Pipeline](https://azure.microsoft.com/en-us/services/devops/pipelines/) that demonstrate end-to-end flow of automated start/deploy/test/publish of integration test using this framework
+* [Python Test Framework](https://docs.python.org/3/library/unittest.html) to perform some basic Integration Test
 
-(Add steps to start up the demo)
+## Expected Outcome
+It is expected to get Integration Test executed from CI Pipeline independently without disturbing existing development. The below images represents the outcome of the [sample for demonstrating the framework](./sample) -
 
-1.
-2.
-3.
+![Sample Azure DevOps Pipeline Execution](docs/images/sample-result.png)
 
 ## Resources
 
-(Any additional resources or related projects)
+Some additional points need to be considered -
 
-- Link to supporting information
-- Link to similar sample
-- ...
+* While using Private Endpoint enabled `Azure Key Vault (AKV)` or `Azure Container Registry (ACR)` for applications, make sure the to use CI Pipeline Agent deployed in the same subnet where `AKV` or `ACR` endpoints are enabled

deploy.sh

@@ -0,0 +1,165 @@
+#/bin/bash
+set +x
+set +e
+
+###########################################################################
+# Usage: bash -f ./deploy.sh
+#        Supported Options - 
+#              --csi-driver-enabled=<yes/no> (default no, if yes provide following two parameters)
+#              --csi-driver-sp-client-id=<Azure Service Principle ID, having access to Azure Key Vault>
+#              --csi-driver-sp-client-secret=<Azure Service Principle Secret, having access to Azure Key Vault>
+#              --acr-imagepullsecret-enabled=<yes/no> (default no, if yes provide following three parameters)
+#              --acr-imagepullsecret-sp-client-id=<Azure Service Principle ID, having access to Azure Container Registry>
+#              --acr-imagepullsecret-sp-client-secret=<Azure Service Principle Secret, having access to Azure Container Registry>
+#              --acr-full-name=<Azure Container Registry full name ex. example.azurecr.io>
+#              --helm-chart-path=<Helm Chart Folder Path or URL to .tgz file for the applications >
+#              --helm-chart-release-name=<Helm Release Name>
+#              --helm-chart-set-parameters=<","(comma) seprated Helm Set parameters needed to be overwritten for integration test env>
+#              --kubectl-check-services=<","(comma) seprated Pod names needed to be check if up and running>
+#              --kubectl-check-services-selector-label=<ex. app.kubernetes.io/name or name etc.> (default app.kubernetes.io/name)
+#              --kubectl-port-forward-services=<","(comma) seprated Service names needed to port-forward for testing>
+###########################################################################
+
+# All Supported Arguments
+ARGUMENT_LIST=(
+   "csi-driver-enabled"
+   "csi-driver-sp-client-id"
+   "csi-driver-sp-client-secret"
+   "acr-imagepullsecret-enabled"
+   "acr-imagepullsecret-sp-client-id"
+   "acr-imagepullsecret-sp-client-secret"
+   "acr-full-name"
+   "helm-chart-path"
+   "helm-chart-release-name"
+   "helm-chart-set-parameters"
+   "kubectl-check-services"
+   "kubectl-check-services-selector-label"
+   "kubectl-port-forward-services"
+)
+
+# Read Arguments
+opts=$(getopt \
+    --longoptions "$(printf "%s:," "${ARGUMENT_LIST[@]}")" \
+    --name "$(basename "$0")" \
+    --options "" \
+    -- "$@"
+)
+
+# Assign Values from Arguments
+eval set --$opts
+while [[ $# -gt 0 ]]; do
+   case "$1" in
+      --csi-driver-enabled)
+         CSI_DRIVER_ENABLED=$(echo "$2" | tr '[:upper:]' '[:lower:]')
+         shift 2
+         ;;
+      --csi-driver-sp-client-id)
+         CSI_DRIVER_SP_CLIENT_ID=$2
+         shift 2
+         ;;
+      --csi-driver-sp-client-secret)
+         CSI_DRIVER_SP_CLIENT_SECRET=$2
+         shift 2
+         ;;
+      --acr-imagepullsecret-enabled)
+         ACR_IMAGEPULLSECRET_ENABLED=$(echo "$2" | tr '[:upper:]' '[:lower:]')
+         shift 2
+         ;;
+      --acr-imagepullsecret-sp-client-id)
+         ACR_IMAGEPULLSECRET_SP_CLIENT_ID=$2
+         shift 2
+         ;;
+      --acr-imagepullsecret-sp-client-secret)
+         ACR_IMAGEPULLSECRET_SP_CLIENT_SECRET=$2
+         shift 2
+         ;;
+      --acr-full-name)
+         ACR_FULL_NAME=$2
+         shift 2
+         ;;
+      --helm-chart-path)
+         HELM_CHART_PATH=$2
+         shift 2
+         ;;
+      --helm-chart-release-name)
+         HELM_CHART_RELEASE_NAME=$2
+         shift 2
+         ;;
+      --helm-chart-set-parameters)
+         HELM_CHART_SET_PARAMETERS=$(echo $2 | sed 's/,\ /,/g')
+         shift 2
+         ;;
+      --kubectl-check-services)
+         KUBECTL_CHECK_SERVICES=$(echo $2 | tr "," "\n")
+         shift 2
+         ;;
+      --kubectl-check-services-selector-label)
+         KUBECTL_CHECK_SERVICES_SELECTOR_LABEL=$2
+         shift 2
+         ;;
+      --kubectl-port-forward-services)
+         KUBECTL_PORT_FORWARD_SERVICES=$(echo $2 | tr "," "\n")
+         shift 2
+         ;;
+      *)
+         break
+         ;;
+   esac
+done
+
+# Assign Deafults
+KUBECTL_CHECK_SERVICES_SELECTOR_LABEL=${KUBECTL_CHECK_SERVICES_SELECTOR_LABEL:-"app.kubernetes.io/name"}
+
+if [[ "${CSI_DRIVER_ENABLED}" == "yes" ]]; then
+   echo $(date -u) "[INFO] Installing CSI Driver ..."
+   linux-amd64/helm repo add csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
+   linux-amd64/helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name --wait
+
+   echo $(date -u) "[INFO] Creating CSI Driver Secret to get Secrets from AKV ..."
+   ./kubectl create secret generic secrets-store-creds \
+      --from-literal clientid="${CSI_DRIVER_SP_CLIENT_ID}" \
+      --from-literal clientsecret="${CSI_DRIVER_SP_CLIENT_SECRET}"
+
+   echo $(date -u) "[INFO] Sleeping 60s to make sure CSI Driver get provisioned .."
+   sleep 60s
+fi
+
+if [[ "${ACR_IMAGEPULLSECRET_ENABLED}" == "yes" ]]; then
+   echo $(date -u) "[INFO] Creating Image Pull Secret ..."
+   ./kubectl create secret docker-registry imagepullsecret \
+      --docker-server="https://${ACR_FULL_NAME}" \
+      --docker-username="${ACR_IMAGEPULLSECRET_SP_CLIENT_ID}" \
+      --docker-password="${ACR_IMAGEPULLSECRET_SP_CLIENT_SECRET}"
+fi
+
+echo $(date -u) "[INFO] Helm install of the Services (timedout in 10 min) ..."
+linux-amd64/helm install ${HELM_CHART_RELEASE_NAME} ${HELM_CHART_PATH} \
+  --wait \
+  --timeout 600s \
+  --set "${HELM_CHART_SET_PARAMETERS}"
+
+echo $(date -u) "[INFO] Sleeping to make sure pods get started .."
+sleep 60s
+
+for _SERVICE in $KUBECTL_CHECK_SERVICES; do
+   echo -n $(date -u) "[INFO] Checking Pods - ${_SERVICE}: "
+   _COUNT=`./kubectl wait --for=condition=ready pod -l ${KUBECTL_CHECK_SERVICES_SELECTOR_LABEL}=${_SERVICE} --timeout=60s| grep met | wc -l`
+   if (( $_COUNT >= 1 )) 
+   then
+      echo "[UP]"
+   else
+      echo "[DOWN]"
+      ./kubectl get pods
+      ./kubectl get events
+   fi
+done
+
+_INDEX=0
+for _SERVICE in $KUBECTL_PORT_FORWARD_SERVICES; do
+   echo $(date -u) "[INFO] Forwarding Service - ${_SERVICE} to http://localhost:808${_INDEX}"
+   ./kubectl port-forward service/${_SERVICE} 808${_INDEX}:80 &
+   let _INDEX=${_INDEX}+1
+done
+
+echo $(date -u) "[INFO] Pod Status"
+./kubectl get pods