additionalUserData.data formatting differences cause drift and Machine recreation

[mmb]

What happened:

When adding additional cloud-init configuration to an RKE2ControlPlane the Machine's rendered RKE2Config has whitespace differences. On a single-node cluster this causes the provider to think the node has drifted and needs a rollout so the Machine gets deleted as soon as the control plane becomes ready. I have not tested with a multi-node control plane but I'm guessing this would cause continuous rollouts.

This only happens with additionalUserData.data, not additionalUserData.config which works as a workaround.

Example additionalUserData inRKE2ControlPlaneTemplate:

---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlaneTemplate
metadata:
  name: test
spec:
  template:
    spec:
      agentConfig:
        additionalUserData:
          data:
            bootcmd: |
              - echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.d/99-custom.conf
              - sysctl --system
            device_aliases:
              data: /dev/nvme0n1
            disk_setup:
              data:
                layout: true
                overwrite: true
                table_type: gpt
              fs_setup:
                - device: data.1
                   filesystem: ext4
                   label: data
              mounts:
                - [ LABEL=data, /data]

This shows the whitespace differences between the RKE2ControlPlane and the RKE2Config:

{"spec"=>
  {"agentConfig"=>
    {"additionalUserData"=>
      {"data"=>
        {"bootcmd"=>
          "- echo \"fs.inotify.max_user_instances=8192\" >> /etc/sysctl.d/99-custom.conf\n" +
          "- sysctl --system\n",
         "device_aliases"=>"data: /dev/nvme0n1\n",
         "disk_setup"=>
          "data:\n" +
          "  layout: true\n" +
          "  overwrite: true\n" +
          "  table_type: gpt\n",
         "fs_setup"=>
          "- device: data.1\n" +
          "  filesystem: ext4\n" +
          "  label: data\n",
         "mounts"=>"- [ LABEL=data, /data ]\n"}}}}}

{"spec"=>
  {"agentConfig"=>
    {"additionalUserData"=>
      {"data"=>
        {"bootcmd"=>
          "\n" +
          "- echo \"fs.inotify.max_user_instances=8192\" >> /etc/sysctl.d/99-custom.conf\n" +
          "- sysctl --system\n",
         "device_aliases"=>"\n" + "  data: /dev/nvme0n1\n" + "  ",
         "disk_setup"=>
          "\n" +
          "  data:\n" +
          "    layout: true\n" +
          "    overwrite: true\n" +
          "    table_type: gpt\n" +
          "  ",
         "fs_setup"=>
          "\n" +
          "- device: data.1\n" +
          "  filesystem: ext4\n" +
          "  label: data\n",
         "mounts"=>"\n" + "- - LABEL=data\n" + "  - /data\n"}}}}}

What did you expect to happen:

There should be no changes so a rollout is not triggered.

How to reproduce it:

Create an RKEControlPlane with additionaUserData.data like above. Bring up a 1-node cluster and see that the machine is deleted after the control plane becomes ready.

Anything else you would like to add:

Environment:

• rke provider version: 0.17.1
• OS (e.g. from /etc/os-release):

[valaparthvi]

@mmb I am surprised you were successfully able to apply the YAML file. When I apply it, I get the following error:

failed to create pvala-ns/pvala-rke2-class-control-plane controlplane.cluster.x-k8s.io/v1beta1, Kind=RKE2ControlPlaneTemplate for a9e49047-0cf5-4b18-ba63-a4edd0c95a19: admission webhook "mrke2controlplanetemplate.kb.io" denied the request: json: cannot unmarshal object into Go struct field AdditionalUserData.spec.template.spec.agentConfig.additionalUserData.data of type string

RKE2ControlPlaneTemplate

kind: RKE2ControlPlaneTemplate
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
metadata:
  name: pvala-rke2-class-control-plane
  namespace: "pvala-ns"
spec:
  template:
    spec:
      agentConfig:
        additionalUserData:
          data:
            bootcmd: |
              - echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.d/99-custom.conf
              - sysctl --system
            device_aliases:
              data: /dev/nvme0n1
            disk_setup:
              data:
                layout: true
                overwrite: true
                table_type: gpt
              fs_setup:
                - device: data.1
                  filesystem: ext4
                  label: data
              mounts:
                - [ LABEL=data, /data ]
      serverConfig:
        cni: calico
        disableComponents:
          kubernetesComponents: [ "cloudController"]
        kubeAPIServer:
          extraArgs:
          - --anonymous-auth=true
      rolloutStrategy:
        type: "RollingUpdate"
        rollingUpdate:
          maxSurge: 1

This error is expected if you look at the expected data type of additionalUserData.data i.e. map[string]string, see here.

I think we would need to fix that part along with the formatting differences. When I tried to simply pass the bootcmd (see below), I could notice the differences between RKE2ControlPlane and RKE2Config.

      agentConfig:
        additionalUserData:
          data:
            bootcmd: |
              - echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.d/99-custom.conf
              - sysctl --system

Diff

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlane
spec:
  agentConfig:
    additionalUserData:
      data:
        bootcmd: >
          - echo "fs.inotify.max_user_instances=8192" >>
          /etc/sysctl.d/99-custom.conf

          - sysctl --system

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: RKE2Config
spec:
  agentConfig:
    additionalUserData:
      data:
        bootcmd: >

          - echo "fs.inotify.max_user_instances=8192" >>
          /etc/sysctl.d/99-custom.conf

          - sysctl --system

RKE provider version: v0.18.0

[mmb]

Yes sorry some of the formatting was lost in my YAML. It should be this:

---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlaneTemplate
metadata:
  name: test
spec:
  template:
    spec:
      agentConfig:
        additionalUserData:
          data:
            bootcmd: |
              - echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.d/99-custom.conf
              - sysctl --system
            device_aliases: |
              data: /dev/nvme0n1
            disk_setup: |
              data:
                layout: true
                overwrite: true
                table_type: gpt
            fs_setup: |
              - device: data.1
                 filesystem: ext4
                 label: data
            mounts: |
              - [ LABEL=data, /data]

The bootcmd alone should be enough of a test case since it shows the difference.