Cleaner separation of kubeadm and machine bootstrapping

[randomvariable]

User Story

As a cluster operator, with development teams requiring the use of multiple operating systems, I would like a better machine bootstrapping abstraction.

Detailed Description

Cluster API Bootstrap Provider Kubeadm currently conflates two activities:

• Generating a kubeadm configuration
• Generating the machine bootstrapping data that eventually executes kubeadm, which only at present supports Cloud Init.

The relationship between Cluster API and machine bootstrapping has created a number of challenges:

How to secure kubeadm node joins

• This will be resolved when secure kubelet authentication is implemented

How to secure control plane instantiation

• Given that to instantiate a control plane on machine boot, we need to give it key material, the CAPI Providers then have to have some mechanism to share this private data with the machine, which has become commonly known as "instance metadata".
• Users which have read only access to the infrastructure can actually read all the private key material and reconstruct administrative access to both etcd and the API server, which represents a privilege escalation risk
• Note that this is separate to kubeadm node joins, and is not resolved with the kubelet authentication plugin proposal
• Some providers have a mechanism to secure the data, i.e. AWS, but these are wholly dependent on the inner workings of cloud-init
  • Specific support needs to be added for each bootstrap mechanism to every cloud provider.

How to extensibly support different bootstrappers without increasing the spaghettiness

• Cluster API currently only supports cloud-init
• PRs are in progress to add support for Ignition v2 (FlatCar) support to CABPK and CAPA (with CAPA's secure instance metadata support)
  • Ignition v3 still will not be supported (required for RedHat/Fedora CoreOS 4.6+ support)
  • Each bootstrapper adds complexity to CABPK and CAPA

Bootstrap reporting

It can be hard to find out what happened when bootstrapping failed. To be fair, the amount of requests for this has gone down over time due to improvements in CABPK and kubeadm, but it's still nice to have ideally.

Anything else you would like to add:

For completeness, and to avoid folk having to work through an unwieldy closed PR, I'm including the user stories and requirements in their entirety from #4221:

User Stories

ID	Title	Description
U1	Non cloud-init bootstrap processes	Ignition is a user-data processing Linux bootstrapping system used by Flat Car Linux, RHEL Atomic Host and Fedora CoreOS. (cluster-api/3761)
U2	System preparation	Although Flatcar Container Linux is being added to Image Builder, Flatcar is intended to also be used as an immutable distribution, with all additions being done at first boot. Flatcar users should be able to use standard Flatcar images with Cluster API.
U3	Active Directory	As a platform operator of a Windows environment, I may require their Kubernetes nodes to be domain joined such that the application workloads operate with appropriate Kerberos credentials to connect to services in the infrastructure. For Windows or Linux hosts joining an Active Directory, they must effectively be given a set of bootstrap credentials to join the directory and persist a Kerberos keytab for the host.
U4	CIS Benchmark Compliance	As a platform operator, I require Kubernetes clusters to pass the CIS Benchmark in order to meet organisational level security compliance requirements.
U5	DISA STIG Compliance	As a platform operator in a US, UK, Canadian, Australian or New Zealand secure government environment, I require my Kubernetes clusters to be compliant with the DISA STIG.
U6	Kubeadm UX	As a cluster operator, I would like the bootstrap configuration of clusters or machines to be shielded from changes happening in kubeadm (e.g. v1beta1 and v1beta2 type migration)
U7	Existing Clusters	As a cluster operator with existing clusters, I would like to be able to, after enabling the necessary flags or feature gates, to create new clusters or machines using nodeadm.
U8	Air-gapped	As a cluster operator, I need Cluster API to operate independently of an internet connection in order to be able to provision clusters in an air-gapped environment, i.e. where the data center is not connected to the public internet.
U9	Advanced control plane configuration files	As a cluster operator, I need to configure components of my control plane, such as audit logging policies, KMS encryption, authentication webhooks to meet organisational requirements.
U10	ContainerD Configuration	Options such as proxy configuration, registry mirrors, custom certs, cgroup hierachy (image-builder/471) need to often be customised, and it isn’t always suitable to do at an image level. Cluster operators in an organisation often resort to prekubeadmcommand bash scripts to configure containerd and restart the service.
U11	API Server Auth Reconfiguration	As a cluster operator, I need to reconfigure the API server such that I can deploy a new static pod for authentication and insert an updated API server configuration.
U12	Improving bootstrap reporting	SRE teams often need to diagnose failed nodes, and having better information about why a node may have failed to join, or better indication of success would be helpful. (cluster-api/3716)
U13	Large payloads	Some vendors, and advanced cluster operators may need to drop large payloads in bootstrap configuration to do a number of tasks, such as drop CA certificates, bootstrap a network components, etc... Cloud providers often have limited sizes for bootstrap data (e.g. AWS/Azure and vSphere)
U14	External bootstrappers	This is to capture the current state that Cluster API allows external bootstrappers to exist, and this should not be changed.

Requirements Specification

We define three modalities of the node bootstrapper:

Mode	Description
Provisioning	Expected to run as part of machine bootstrapping e.g. (part of cloud-* SystemD units or Windows OOBE). Only supported when used with Cluster API bootstrapping. Typically executes cluster creation or node join procedures, configuring kubelet etc...
Preparation	Could be run as part of machine bootstrapping prior to “provisioning”, and “prepares” a machine for use with Kubernetes. We largely keep this out of scope for the initial implementation unless there is a trivial implementation.
Post	Parts of the use cases above require ongoing management of a host. We list these as requirements, but are largely not in scope for the machine bootstrapper and should be dealt with by external systems.

ID	Requirement	Mode	Related Stories
R1	The machine bootstrapper MUST be able to execute kubeadm and report its outcome. Provisioning	Provisioning	U1
R2	The machine bootstrapper MUST allow the configuration of Linux sysctl parameters	Preparation	U2,U4
R3	The machine bootstrapper COULD allow the application of custom static pods on the control plane	Provisioning	U4,U9
R4	The machine bootstrapper MUST not directly expose the kubeadm API to the end user	Provisioning	U6
R5	The machine bootstrapper MUST be able to be used in conjunction with an OS provided bootstrapping tool, not limited to Cloud-Init, Ignition, Talos and Windows Answer File.	Provisioning	U1
R6	The machine bootstrapper/authenticator binary MUST provide cryptographic verification in situations where it is downloaded post-boot.	Preparation	U2
R7	The machine bootstrapper MUST not be reliant on the use of static pods to operate	All	U5
R8	The machine bootstrapper MUST enable a Windows node to be domain joined. The machine bootstrapper WILL NOT manage the group membership of a Windows node in order to enable Group Managed Service Accounts	Provisioning	U3
R9	The node bootstrapping system MUST be opt-in and not affect the operation of existing clusters when Cluster API is upgraded.	Provisioning	U7
R10	The machine bootstrapper system SHOULD allow the agent to be downloaded from the management cluster	Preparation	U8
R11	The machine bootstrapper MUST be able to operate without connectivity to the internet (using proper configuration parameters), or to the management cluster.	Provisioning	U7
R12	When the machine bootstrapper is downloaded on boot the location MUST be configurable	Preparation	U8
R13	When the machine bootstrapper is downloaded from the public internet, it MUST be downloadable from a location not subject to frequent rate limiting (e.g. a GCS bucket).	Preparation	U9
R14	The machine bootstrapper MUST be able to configure containerd given a structured configuration input..	Provisioning	U10
R15	The machine bootstrapper MUST publish a documented contract for operating system maintainers to integrate with the machine bootstrapper.	All	U1
R16	The machine bootstrapper MUST support pulling payloads from a defined location outside of the cloud provider's Instance Metadata Service in order to cope with large payloads.	All	U13
R17	The machine bootstrapper MUST not preclude the use of external bootstrappers as is the case today.	All	U14

/kind feature

An example of the current flow for AWS is here (courtesy of @PushkarJ )

[vincepri]

/assign @killianmuldoon

[randomvariable]

One excellent suggestion from the lengthy discussion in that proposal was that we should https://github.com/mozilla/sops as the encryption envelope for private key material.

[CecileRobertMichon]

/cc @t-lo

[JoelSpeed]

Just wanted to clarify what we are talking about here, is the purpose of this issue to define an interface for various providers to implement (for different OSs?) or to define a tool that will be implemented?

Reading through some of this it seemed as if this is talking about building some bootstrap binary that would allow configuration of various OSs, but I initially had assumed this would define an interface

[randomvariable]

Reading through some of this it seemed as if this is talking about building some bootstrap binary that would allow configuration of various OSs, but I initially had assumed this would define an interface

I think it would be both: An interface with a default implementation.

[vincepri]

/milestone v1.0

[vincepri]

/kind proposal

[randomvariable]

cc @richardcase @codablock

I've been reviewing some PRs in CAPA, namely kubernetes-sigs/cluster-api-provider-aws#2854 and it looks like EKS has the same challenges for some areas, e.g. User Story 10, and the way it's being tackled there is to add shell scripts to the EKS image builder equivalent and then add an API in CAPA. I wonder if we should consider this as a new project and get some folk together.

[richardcase]

I've been reviewing some PRs in CAPA, namely kubernetes-sigs/cluster-api-provider-aws#2854 and it looks like EKS has the same challenges for some areas, e.g. User Story 10, and the way it's being tackled there is to add shell scripts to the EKS image builder equivalent and then add an API in CAPA. I wonder if we should consider this as a new project and get some folk together.

It does have some of the same challenges for sure. And yes it would be good to get some people together to start discussions on this.