Merge pull request #2 from prabdeb/cosmetic-changes

First version 1.0.0

Author: prabdeb

.github/ISSUE_TEMPLATE.md

@@ -21,7 +21,7 @@ IF SUFFICIENT INFORMATION IS NOT PROVIDED VIA THE FOLLOWING TEMPLATE THE ISSUE M
 ### Expected/desired behavior
 >
 
-### OS and Version?
+### Pipeline Agent?
 > Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
 
 ### Versions

.gitignore

@@ -20,8 +20,6 @@ parts/
 sdist/
 var/
 wheels/
-pip-wheel-metadata/
-share/python-wheels/
 *.egg-info/
 .installed.cfg
 *.egg
@@ -40,14 +38,12 @@ pip-delete-this-directory.txt
 # Unit test / coverage reports
 htmlcov/
 .tox/
-.nox/
 .coverage
 .coverage.*
 .cache
 nosetests.xml
 coverage.xml
 *.cover
-*.py,cover
 .hypothesis/
 .pytest_cache/
 
@@ -59,7 +55,6 @@ coverage.xml
 *.log
 local_settings.py
 db.sqlite3
-db.sqlite3-journal
 
 # Flask stuff:
 instance/
@@ -77,26 +72,11 @@ target/
 # Jupyter Notebook
 .ipynb_checkpoints
 
-# IPython
-profile_default/
-ipython_config.py
-
 # pyenv
 .python-version
 
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow
-__pypackages__/
-
-# Celery stuff
+# celery beat schedule file
 celerybeat-schedule
-celerybeat.pid
 
 # SageMath parsed files
 *.sage.py
@@ -122,8 +102,9 @@ venv.bak/
 
 # mypy
 .mypy_cache/
-.dmypy.json
-dmypy.json
 
-# Pyre type checker
-.pyre/
+# Framework related
+kubectl
+kind
+helm*
+linux-amd64/*

CHANGELOG.md

@@ -1,13 +1,19 @@
-## [project-title] Changelog
+## Automated Test Environment for AKS Applications Changelog
 
-<a name="x.y.z"></a>
-# x.y.z (yyyy-mm-dd)
+<a name="1.0.0"></a>
+# 1.0.0 (2020-08-03)
 
 *Features*
-* ...
+* Create/Delete `kind` cluster in CI environment
+* Optional - [Azure Key Vault Provider for Secrets Store CSI Driver](https://github.com/Azure/secrets-store-csi-driver-provider-azure) [installation](https://github.com/Azure/secrets-store-csi-driver-provider-azure#install-the-secrets-store-csi-driver-and-the-azure-keyvault-provider) and [configuration](https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/docs/service-principal-mode.md)
+* Optional - [Azure Container Registry (ACR) Image Pull Secret](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-kubernetes#create-an-image-pull-secret)
+* Install `helm` charts of application (values needed for integration test environment can be overwritten easily)
+* Validate if the respective Kubernetes `pods` are up and running (multiple `pods` can be provided that are having selector - `app.kubernetes.io/name`)
+* [Port-Froward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/#forward-a-local-port-to-a-port-on-the-pod) the respective Kubernetes `services` needed to perform integration testing (multiple `services` can be provided and the respective local ports will be starting from `8080` to `808[number of services]` maintaining the order as provided)
+
 
 *Bug Fixes*
-* ...
+* N/A
 
 *Breaking Changes*
-* ...
+* N/A

README.md

@@ -1,57 +1,81 @@
-# Project Name
+# Automated Test Environment for AKS Applications
 
-(short, 1-3 sentenced, description of the project)
+Framework to create Automated Test Environment using [KIND](https://kind.sigs.k8s.io/) for testing Azure Kubernetes Service (AKS) based applications in CI (Continuous Integration) Pipeline (Azure DevOps), where required dependencies/infrastructure will be provisioned for executing automated tests and deprovisioned after completion.
 
-## Features
-
-This project framework provides the following features:
-
-* Feature 1
-* Feature 2
-* ...
-
-## Getting Started
+Testing the applications to be deployed over Kubernetes means either the Cluster is already available or is to be deployed on the go before you test, in former case its a cost to keep the Cluster live and in later it takes a lot of time to bring up the kubernetes cluster.
 
-### Prerequisites
+ This framework uses [KIND](https://kind.sigs.k8s.io/) which itself was primarily designed for testing Kubernetes itself. KIND is often used by developers to test their applications in local dev environments and can very well be used for automated testing.
 
-(ideally very short, if any)
+## Tools Used
+1. [KIND](https://kind.sigs.k8s.io/) - a tool for running local Kubernetes clusters using Docker container “nodes”
+2. [helm](https://helm.sh/) - the package manager for Kubernetes
+3. [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) - command line tool to control Kubernetes clusters
+4. [bash](https://www.gnu.org/software/bash/) - the GNU Project's shell
 
-- OS
-- Library version
-- ...
-
-### Installation
-
-(ideally very short)
+## Features
 
-- npm install [package name]
-- mvn install
-- ...
+This is a very simple Framework, that creates Automated Test Environment to enable automated testing of applications hosted in [Azure Kubernetes Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/) with the following features -
 
-### Quickstart
-(Add steps to get up and running quickly)
+1. Create/Delete `KIND` cluster in CI environment
+2. Optional - [Azure Key Vault Provider for Secrets Store CSI Driver](https://github.com/Azure/secrets-store-csi-driver-provider-azure) [installation](https://github.com/Azure/secrets-store-csi-driver-provider-azure#install-the-secrets-store-csi-driver-and-the-azure-keyvault-provider) and [configuration](https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/docs/service-principal-mode.md)
+3. Optional - [Azure Container Registry (ACR) Image Pull Secret](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-kubernetes#create-an-image-pull-secret)
+4. Install `helm` charts of application (values needed for integration test environment can be overwritten easily)
+5. Validate if the respective Kubernetes `pods` are up and running (multiple `pods` can be provided that are having selector - `app.kubernetes.io/name`)
+6. [Port-Froward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/#forward-a-local-port-to-a-port-on-the-pod) the respective Kubernetes `services` needed to perform integration testing (multiple `services` can be provided and the respective local ports will be starting from `8080` to `808[number of services]` maintaining the order as provided)
 
-1. git clone [repository clone url]
-2. cd [respository name]
-3. ...
+## Getting Started
 
+This framework contains the following script and their options -
+
+1. `start.sh`: Download all the dependencies and create `kind` cluster -
+```sh
+# Usage: bash -f ./start.sh
+#        Supported Options -
+#               --kind-cluster-name=<Kind Cluster Name> (default INTEGRATION_TEST_CLUSTER)
+#               --kind-version (default v0.7.0)
+#               --kubectl-version (default v1.18.0)
+#               --helm-version (default v3.2.0)
+```
+2. `deploy.sh`: Deploy/Port-Forward application `helm chart` and enables optional features -
+```sh
+# Usage: bash -f ./deploy.sh
+#        Supported Options - 
+#              --csi-driver-enabled=<yes/no> (default no, if yes provide following two parameters)
+#              --csi-driver-sp-client-id=<Azure Service Principle ID, having access to Azure Key Vault>
+#              --csi-driver-sp-client-secret=<Azure Service Principle Secret, having access to Azure Key Vault>
+#              --acr-imagepullsecret-enabled=<yes/no> (default no, if yes provide following three parameters)
+#              --acr-imagepullsecret-sp-client-id=<Azure Service Principle ID, having access to Azure Container Registry>
+#              --acr-imagepullsecret-sp-client-secret=<Azure Service Principle Secret, having access to Azure Container Registry>
+#              --acr-full-name=<Azure Container Registry full name ex. example.azurecr.io>
+#              --helm-chart-path=<Helm Chart Folder Path or URL to .tgz file for the applications >
+#              --helm-chart-release-name=<Helm Release Name>
+#              --helm-chart-set-parameters=<","(comma) seprated Helm Set parameters needed to be overwritten for integration test env>
+#              --kubectl-check-services=<","(comma) seprated Pod names needed to be check if up and running>
+#              --kubectl-check-services-selector-label=<ex. app.kubernetes.io/name or name etc.> (default app.kubernetes.io/name)
+#              --kubectl-port-forward-services=<","(comma) seprated Service names needed to port-forward for testing>
+```
+3. `stop.sh`: Delete `KIND` cluster -
+```sh
+# Usage: bash -f ./stop.sh
+#        Supported Options -
+#               --kind-cluster-name=<Kind Cluster Name> (default INTEGRATION_TEST_CLUSTER)
+```
 
 ## Demo
 
-A demo app is included to show how to use the project.
+For [demonstrating the framework](./sample) following setup has been used -
 
-To run the demo, follow these steps:
+* Sample Application [azure-vote](https://github.com/Azure-Samples/helm-charts/tree/master/chart-source/azure-vote)
+* [Azure DevOps Pipeline](https://azure.microsoft.com/en-us/services/devops/pipelines/) that demonstrate end-to-end flow of automated start/deploy/test/publish of integration test using this framework
+* [Python Test Framework](https://docs.python.org/3/library/unittest.html) to perform some basic Integration Test
 
-(Add steps to start up the demo)
+## Expected Outcome
+It is expected to get Integration Test executed from CI Pipeline independently without disturbing existing development. The below images represents the outcome of the [sample for demonstrating the framework](./sample) -
 
-1.
-2.
-3.
+![Sample Azure DevOps Pipeline Execution](docs/images/sample-result.png)
 
 ## Resources
 
-(Any additional resources or related projects)
+Some additional points need to be considered -
 
-- Link to supporting information
-- Link to similar sample
-- ...
+* While using Private Endpoint enabled `Azure Key Vault (AKV)` or `Azure Container Registry (ACR)` for applications, make sure the to use CI Pipeline Agent deployed in the same subnet where `AKV` or `ACR` endpoints are enabled

deploy.sh

@@ -0,0 +1,165 @@
+#/bin/bash
+set +x
+set +e
+
+###########################################################################
+# Usage: bash -f ./deploy.sh
+#        Supported Options - 
+#              --csi-driver-enabled=<yes/no> (default no, if yes provide following two parameters)
+#              --csi-driver-sp-client-id=<Azure Service Principle ID, having access to Azure Key Vault>
+#              --csi-driver-sp-client-secret=<Azure Service Principle Secret, having access to Azure Key Vault>
+#              --acr-imagepullsecret-enabled=<yes/no> (default no, if yes provide following three parameters)
+#              --acr-imagepullsecret-sp-client-id=<Azure Service Principle ID, having access to Azure Container Registry>
+#              --acr-imagepullsecret-sp-client-secret=<Azure Service Principle Secret, having access to Azure Container Registry>
+#              --acr-full-name=<Azure Container Registry full name ex. example.azurecr.io>
+#              --helm-chart-path=<Helm Chart Folder Path or URL to .tgz file for the applications >
+#              --helm-chart-release-name=<Helm Release Name>
+#              --helm-chart-set-parameters=<","(comma) seprated Helm Set parameters needed to be overwritten for integration test env>
+#              --kubectl-check-services=<","(comma) seprated Pod names needed to be check if up and running>
+#              --kubectl-check-services-selector-label=<ex. app.kubernetes.io/name or name etc.> (default app.kubernetes.io/name)
+#              --kubectl-port-forward-services=<","(comma) seprated Service names needed to port-forward for testing>
+###########################################################################
+
+# All Supported Arguments
+ARGUMENT_LIST=(
+   "csi-driver-enabled"
+   "csi-driver-sp-client-id"
+   "csi-driver-sp-client-secret"
+   "acr-imagepullsecret-enabled"
+   "acr-imagepullsecret-sp-client-id"
+   "acr-imagepullsecret-sp-client-secret"
+   "acr-full-name"
+   "helm-chart-path"
+   "helm-chart-release-name"
+   "helm-chart-set-parameters"
+   "kubectl-check-services"
+   "kubectl-check-services-selector-label"
+   "kubectl-port-forward-services"
+)
+
+# Read Arguments
+opts=$(getopt \
+    --longoptions "$(printf "%s:," "${ARGUMENT_LIST[@]}")" \
+    --name "$(basename "$0")" \
+    --options "" \
+    -- "$@"
+)
+
+# Assign Values from Arguments
+eval set --$opts
+while [[ $# -gt 0 ]]; do
+   case "$1" in
+      --csi-driver-enabled)
+         CSI_DRIVER_ENABLED=$(echo "$2" | tr '[:upper:]' '[:lower:]')
+         shift 2
+         ;;
+      --csi-driver-sp-client-id)
+         CSI_DRIVER_SP_CLIENT_ID=$2
+         shift 2
+         ;;
+      --csi-driver-sp-client-secret)
+         CSI_DRIVER_SP_CLIENT_SECRET=$2
+         shift 2
+         ;;
+      --acr-imagepullsecret-enabled)
+         ACR_IMAGEPULLSECRET_ENABLED=$(echo "$2" | tr '[:upper:]' '[:lower:]')
+         shift 2
+         ;;
+      --acr-imagepullsecret-sp-client-id)
+         ACR_IMAGEPULLSECRET_SP_CLIENT_ID=$2
+         shift 2
+         ;;
+      --acr-imagepullsecret-sp-client-secret)
+         ACR_IMAGEPULLSECRET_SP_CLIENT_SECRET=$2
+         shift 2
+         ;;
+      --acr-full-name)
+         ACR_FULL_NAME=$2
+         shift 2
+         ;;
+      --helm-chart-path)
+         HELM_CHART_PATH=$2
+         shift 2
+         ;;
+      --helm-chart-release-name)
+         HELM_CHART_RELEASE_NAME=$2
+         shift 2
+         ;;
+      --helm-chart-set-parameters)
+         HELM_CHART_SET_PARAMETERS=$(echo $2 | sed 's/,\ /,/g')
+         shift 2
+         ;;
+      --kubectl-check-services)
+         KUBECTL_CHECK_SERVICES=$(echo $2 | tr "," "\n")
+         shift 2
+         ;;
+      --kubectl-check-services-selector-label)
+         KUBECTL_CHECK_SERVICES_SELECTOR_LABEL=$2
+         shift 2
+         ;;
+      --kubectl-port-forward-services)
+         KUBECTL_PORT_FORWARD_SERVICES=$(echo $2 | tr "," "\n")
+         shift 2
+         ;;
+      *)
+         break
+         ;;
+   esac
+done
+
+# Assign Deafults
+KUBECTL_CHECK_SERVICES_SELECTOR_LABEL=${KUBECTL_CHECK_SERVICES_SELECTOR_LABEL:-"app.kubernetes.io/name"}
+
+if [[ "${CSI_DRIVER_ENABLED}" == "yes" ]]; then
+   echo $(date -u) "[INFO] Installing CSI Driver ..."
+   linux-amd64/helm repo add csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
+   linux-amd64/helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name --wait
+
+   echo $(date -u) "[INFO] Creating CSI Driver Secret to get Secrets from AKV ..."
+   ./kubectl create secret generic secrets-store-creds \
+      --from-literal clientid="${CSI_DRIVER_SP_CLIENT_ID}" \
+      --from-literal clientsecret="${CSI_DRIVER_SP_CLIENT_SECRET}"
+
+   echo $(date -u) "[INFO] Sleeping 60s to make sure CSI Driver get provisioned .."
+   sleep 60s
+fi
+
+if [[ "${ACR_IMAGEPULLSECRET_ENABLED}" == "yes" ]]; then
+   echo $(date -u) "[INFO] Creating Image Pull Secret ..."
+   ./kubectl create secret docker-registry imagepullsecret \
+      --docker-server="https://${ACR_FULL_NAME}" \
+      --docker-username="${ACR_IMAGEPULLSECRET_SP_CLIENT_ID}" \
+      --docker-password="${ACR_IMAGEPULLSECRET_SP_CLIENT_SECRET}"
+fi
+
+echo $(date -u) "[INFO] Helm install of the Services (timedout in 10 min) ..."
+linux-amd64/helm install ${HELM_CHART_RELEASE_NAME} ${HELM_CHART_PATH} \
+  --wait \
+  --timeout 600s \
+  --set "${HELM_CHART_SET_PARAMETERS}"
+
+echo $(date -u) "[INFO] Sleeping to make sure pods get started .."
+sleep 60s
+
+for _SERVICE in $KUBECTL_CHECK_SERVICES; do
+   echo -n $(date -u) "[INFO] Checking Pods - ${_SERVICE}: "
+   _COUNT=`./kubectl wait --for=condition=ready pod -l ${KUBECTL_CHECK_SERVICES_SELECTOR_LABEL}=${_SERVICE} --timeout=60s| grep met | wc -l`
+   if (( $_COUNT >= 1 )) 
+   then
+      echo "[UP]"
+   else
+      echo "[DOWN]"
+      ./kubectl get pods
+      ./kubectl get events
+   fi
+done
+
+_INDEX=0
+for _SERVICE in $KUBECTL_PORT_FORWARD_SERVICES; do
+   echo $(date -u) "[INFO] Forwarding Service - ${_SERVICE} to http://localhost:808${_INDEX}"
+   ./kubectl port-forward service/${_SERVICE} 808${_INDEX}:80 &
+   let _INDEX=${_INDEX}+1
+done
+
+echo $(date -u) "[INFO] Pod Status"
+./kubectl get pods