Merge pull request #1801 from oracle-devrel/oke-rm

Oke resource manager stack v1.1.0

Author: martatolosa

.gitignore

@@ -3,3 +3,34 @@ shared-assets/bastion-py-script/.oci/
 shared-assets/bastion-py-script/temp/
 temp/
 app-dev/app-integration-and-automation/oracle-integration-cloud/01-oic-connectivity-agent/README_tmp.html
+
+
+# Local .terraform directories
+**/.terraform/*
+
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Exclude all .tfvars files, which are likely to contain sensitive data, such as
+# password, private keys, and other secrets. These should not be part of version
+# control as they are data points which are potentially sensitive and subject
+# to change depending on the environment.
+*.tfvars
+*.tfvars.json
+
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
+
+# Terraform lock files
+*.lock.hcl

app-dev/devops-and-containers/oke/oke-rm/README.md

@@ -0,0 +1,81 @@
+# OKE Quickstart
+
+This repository was created with the intent of facilitating users with the creation of an OKE cluster from scratch.
+
+The plan is to have a documentation and some stacks for the majority of use cases.
+
+In this repository we are going to provision all the components one by one (network, OKE control plane, OKE data plane)
+
+NOTE: If you want to create an OKE cluster with GPU and RDMA, then the stack that creates everything is public and available [here](https://github.com/oracle-quickstart/oci-hpc-oke)
+
+## Step 1: Create the network infrastructure for OKE
+
+This stack is used to create the initial network infrastructure for OKE. When configuring it, pay attention to some details:
+* Select Flannel as CNI if you are planning to use Bare Metal shapes for the OKE data plane, or if you do not have many IPs available in the VCN
+* You can apply this stack even on an existing VCN, so that only the NSGs for OKE will be created
+* By default, everything is private, but there is the possibility to create public subnets
+* Be careful when modifying the default values, as inputs are not validated
+
+[![Deploy to Oracle Cloud](https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg)](https://cloud.oracle.com/resourcemanager/stacks/create?zipUrl=https://github.com/oracle-devrel/technology-engineering/releases/download/oke-rm-1.1.0/infra.zip)
+
+## Step 2: Create the OKE control plane
+
+This stack is used to create the OKE control plane ONLY.
+
+[![Deploy to Oracle Cloud](https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg)](https://cloud.oracle.com/resourcemanager/stacks/create?zipUrl=https://github.com/oracle-devrel/technology-engineering/releases/download/oke-rm-1.1.0/oke.zip)
+
+Also note that if the network infrastructure is located in a different compartment than the OKE cluster AND you are planning to use the OCI_VCN_NATIVE CNI,
+you must add these policies:
+
+```ignorelang
+Allow any-user to manage instances in tenancy where all { request.principal.type = 'cluster' }
+Allow any-user to use private-ips in tenancy where all { request.principal.type = 'cluster' }
+Allow any-user to use network-security-groups in tenancy where all { request.principal.type = 'cluster' }
+```
+For a more restrictive set of policies, see the [documentation](https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking_topic-OCI_CNI_plugin.htm).
+
+## Step 3: Create the OKE data plane
+
+As the data plane vastly depends on the particular use case, there is no stack for it, as there are many options.
+
+### Option 3.1: Create the OKE data plane with Oracle Linux nodes
+
+This option is most commonly used for general purpose CPU workloads.
+
+Although GPU workloads are supported too, the Nvidia GPU Operator is not supported, so take this into account if you are planning to use Oracle Linux nodes and GPUs.
+
+#### Option 3.1.1: Create worker nodes manually through the OCI web console
+
+In some cases, some users prefer to create the nodes directly using the OCI web console. In this case there is nothing else to do, you are free to login and create the node pools.
+
+#### Option 3.1.2: Create worker nodes by modifying the Terraform Resource Manager stack
+
+It is possible to easily modify the Terraform code of an OCI Resource Manager stack.
+
+By using this feature, we can modify the stack we deployed in Step 2 and add the data plane nodes:
+
+![Edit Terraform configurations](images/edit_oci_stack.png)
+
+Instructions on how to modify the stack and add node pools can be found in the comments of the oke.tf file.
+
+### Option 3.2: Create the OKE data plane with Ubuntu nodes
+
+This option is most commonly used for AI workloads and GPU nodes, as Nvidia officially supports the Nvidia GPU plugin and DCGM exporter only on Ubuntu.
+
+#### Option 3.2.1: Create worker nodes by modifying the Terraform Resource Manager stack
+
+To use Ubuntu nodes on OKE, an Ubuntu custom image must be created beforehand. Documentation on how to do this is present in the oke.tf comments.
+
+Once we have an image, we can modify the Terraform configurations directly from the OCI web console, as with option 3.1.2
+
+### Option 3.3: Create an OKE RDMA cluster with Ubuntu nodes
+
+If you are looking to provision an OKE cluster for RDMA and GPUs using this stack and approach, feel free to contact one of the [EMEA AppDev team](../../../README.md) as we prefer to help you, and to give you some tips to go faster.
+
+# What to do next?
+
+Provisioning an OKE cluster is just the first step, be sure to also check out these guides to learn how to configure it:
+* [OKE policies](../oke-policies/policies.md)
+* [GitOps with ArgoCD](https://github.com/alcampag/oke-gitops)
+* [Ingress guide](ingress.md)
+

app-dev/devops-and-containers/oke/oke-rm/images/edit_oci_stack.png

@@ -0,0 +1,3 @@
+locals {
+  create_bastion = var.create_bastion_subnet && var.create_bastion
+}

app-dev/devops-and-containers/oke/oke-rm/infra/infra.zip

@@ -0,0 +1,66 @@
+
+module "network" {
+  source = "./modules/network"
+  network_compartment_id = var.network_compartment_id
+  region = var.region
+  cni_type = var.cni_type
+  # VCN
+  create_vcn = var.create_vcn
+  vcn_id = var.vcn_id
+  vcn_name = var.vcn_name
+  vcn_cidr_blocks = var.vcn_cidr_blocks
+  vcn_dns_label = var.vcn_dns_label
+  # CP SUBNET
+  create_cp_subnet = var.create_cp_subnet
+  cp_subnet_cidr = var.cp_subnet_cidr
+  cp_subnet_dns_label = var.cp_subnet_dns_label
+  cp_subnet_name = var.cp_subnet_name
+  cp_subnet_private = var.cp_subnet_private
+  cp_allowed_source_cidr = var.cp_allowed_source_cidr
+  # SERVICE SUBNET
+  create_service_subnet = var.create_service_subnet
+  service_subnet_cidr = var.service_subnet_cidr
+  service_subnet_dns_label = var.service_subnet_dns_label
+  service_subnet_name = var.service_subnet_name
+  service_subnet_private = var.service_subnet_private
+  # WORKER SUBNET
+  create_worker_subnet = var.create_worker_subnet
+  worker_subnet_cidr = var.worker_subnet_cidr
+  worker_subnet_dns_label = var.worker_subnet_dns_label
+  worker_subnet_name = var.worker_subnet_name
+  # POD SUBNET
+  create_pod_subnet = var.create_pod_subnet
+  pod_subnet_cidr = var.pod_subnet_cidr
+  pod_subnet_dns_label = var.pod_subnet_dns_label
+  pod_subnet_name = var.pod_subnet_name
+  # BASTION SUBNET
+  create_bastion_subnet = var.create_bastion_subnet
+  bastion_subnet_cidr = var.bastion_subnet_cidr
+  bastion_subnet_dns_label = var.bastion_subnet_dns_label
+  bastion_subnet_name = var.bastion_subnet_name
+  bastion_subnet_private = var.bastion_subnet_private
+  # FSS SUBNET
+  create_fss = var.create_fss
+  fss_subnet_cidr = var.fss_subnet_cidr
+  fss_subnet_dns_label = var.fss_subnet_dns_label
+  fss_subnet_name = var.fss_subnet_name
+  # GATEWAYS
+  create_gateways = var.create_gateways
+  nat_gateway_id = var.nat_gateway_id
+  service_gateway_id = var.service_gateway_id
+  create_internet_gateway = var.create_internet_gateway
+  # CONTROL PLANE EXTERNAL CONNECTION
+  cp_external_nat = var.cp_external_nat
+  allow_external_cp_traffic = var.allow_external_cp_traffic
+  cp_egress_cidr = var.cp_egress_cidr
+}
+
+module "bastion" {
+  source = "./modules/bastion"
+  region = var.region
+  compartment_id = var.bastion_compartment_id
+  vcn_name = var.vcn_name
+  bastion_subnet_id = module.network.bastion_subnet_id
+  bastion_cidr_block_allow_list = var.bastion_cidr_block_allow_list
+  count = local.create_bastion ? 1 : 0
+}

app-dev/devops-and-containers/oke/oke-rm/infra/local.tf

@@ -0,0 +1,8 @@
+resource "oci_bastion_bastion" "vcn_spoke_bastion" {
+  bastion_type     = "STANDARD"
+  compartment_id   = var.compartment_id
+  target_subnet_id = var.bastion_subnet_id
+  name = "bastion-${var.vcn_name}"
+  dns_proxy_status = "ENABLED"
+  client_cidr_block_allow_list = var.bastion_cidr_block_allow_list
+}

app-dev/devops-and-containers/oke/oke-rm/infra/main.tf

@@ -0,0 +1,8 @@
+terraform {
+  required_providers {
+    oci = {
+      source  = "oracle/oci"
+      version = ">= 6.0.0"
+    }
+  }
+}

app-dev/devops-and-containers/oke/oke-rm/infra/modules/bastion/bastion.tf

@@ -0,0 +1,9 @@
+variable "region" {}
+variable "compartment_id" {}
+variable "bastion_subnet_id" {}
+variable "vcn_name" {}
+
+
+variable "bastion_cidr_block_allow_list" {
+  type = list(string)
+}

app-dev/devops-and-containers/oke/oke-rm/infra/modules/bastion/provider.tf

@@ -0,0 +1,22 @@
+resource "oci_core_security_list" "bastion_security_list" {
+  compartment_id = var.network_compartment_id
+  vcn_id         = local.vcn_id
+  display_name = "bastion-sec-list"
+  ingress_security_rules {
+    protocol = "6"
+    source_type = "CIDR_BLOCK"
+    source   = "0.0.0.0/0"
+    description = "Allow SSH connections to the subnet. Can be deleted if only using OCI Bastion subnet"
+    tcp_options {
+      max = 22
+      min = 22
+    }
+  }
+  egress_security_rules {
+    destination = var.vcn_cidr_blocks[0]
+    destination_type = "CIDR_BLOCK"
+    protocol    = "all"
+    description = "Enable the bastion hosts to reach the entire VCN"
+  }
+  count = var.create_bastion_subnet ? 1 : 0
+}