Skip to content

Feature: Disable public ips on autoscaled nodes #1695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1013,6 +1013,7 @@ If you follow this values, in your kube.tf, please set:
- `existing_network_id = [YOURID]` (with the brackets)
- `network_ipv4_cidr = "10.0.0.0/9"`
- Add `disable_ipv4 = true` and `disable_ipv6 = true` in all machines in all nodepools (control planes + agents).
- Add `autoscaler_disable_ipv4 = true` and `autoscaler_disable_ipv6 = true` to disable public ips on autoscaled nodes.

This setup is compatible with a loadbalancer for your control planes, however you should consider to set
`control_plane_lb_enable_public_interface = false` to keep ip private.
Expand Down
10 changes: 7 additions & 3 deletions autoscaler-agents.tf
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ locals {
cluster_config = base64encode(jsonencode(local.cluster_config))
firewall_id = hcloud_firewall.k3s.id
cluster_name = local.cluster_prefix
node_pools = var.autoscaler_nodepools
node_pools = var.autoscaler_nodepools,
disable_ipv4 = var.autoscaler_disable_ipv4,
disable_ipv6 = var.autoscaler_disable_ipv6,
Comment on lines +44 to +46
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Consider adding a comment explaining why these variables are being passed directly into the local block. This will help future maintainers understand the purpose of these variables and how they affect the autoscaler configuration.

      node_pools                                 = var.autoscaler_nodepools,
      # Pass disable_ipv4 and disable_ipv6 to configure network settings for autoscaled nodes
      disable_ipv4                               = var.autoscaler_disable_ipv4,
      disable_ipv6                               = var.autoscaler_disable_ipv6,

})
# A concatenated list of all autoscaled nodes
autoscaled_nodes = length(var.autoscaler_nodepools) == 0 ? {} : {
Expand Down Expand Up @@ -116,7 +118,8 @@ data "cloudinit_config" "autoscaler_config" {
})
install_k3s_agent_script = join("\n", concat(local.install_k3s_agent, ["systemctl start k3s-agent"]))
cloudinit_write_files_common = local.cloudinit_write_files_common
cloudinit_runcmd_common = local.cloudinit_runcmd_common
cloudinit_runcmd_common = local.cloudinit_runcmd_common,
private_network_only = var.autoscaler_disable_ipv4 && var.autoscaler_disable_ipv6,
}
)
}
Expand Down Expand Up @@ -150,7 +153,8 @@ data "cloudinit_config" "autoscaler_legacy_config" {
})
install_k3s_agent_script = join("\n", concat(local.install_k3s_agent, ["systemctl start k3s-agent"]))
cloudinit_write_files_common = local.cloudinit_write_files_common
cloudinit_runcmd_common = local.cloudinit_runcmd_common
cloudinit_runcmd_common = local.cloudinit_runcmd_common,
private_network_only = var.autoscaler_disable_ipv4 && var.autoscaler_disable_ipv6,
}
)
}
Expand Down
6 changes: 5 additions & 1 deletion kube.tf.example
Original file line number Diff line number Diff line change
Expand Up @@ -360,6 +360,10 @@ module "kube-hetzner" {
# # kubelet_args = ["kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi", "system-reserved=cpu=250m,memory=300Mi"]
# }
# ]
#
# To disable public ips on your autoscaled nodes, uncomment the following lines:
# autoscaler_disable_ipv4 = true
# autoscaler_disable_ipv6 = true

# ⚠️ Deprecated, will be removed after a new Cluster Autoscaler version has been released which support the new way of setting labels and taints. See above.
# Add extra labels on nodes started by the Cluster Autoscaler
Expand Down Expand Up @@ -1169,7 +1173,7 @@ terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = ">= 1.49.1"
version = ">= 1.51.0"
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion modules/host/versions.tf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = ">= 1.49.1"
version = ">= 1.51.0"
}
}
}
7 changes: 7 additions & 0 deletions templates/autoscaler-cloudinit.yaml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,12 @@ runcmd:

${cloudinit_runcmd_common}

# Configure default route based on public ip availability
%{if private_network_only~}
- [ip, route, add, default, via, '10.0.0.1', dev, 'eth0']
%{else~}
- [ip, route, add, default, via, '172.31.1.1', dev, 'eth0']
%{endif~}
Comment on lines +45 to +50
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This section configures the default route based on private_network_only. It's crucial to ensure that this logic is robust and handles all possible scenarios correctly. Consider adding comments to explain the IP addresses used (10.0.0.1 and 172.31.1.1) and their significance in the context of Hetzner Cloud networking. What happens if neither of these routes are valid?

# Configure default route based on public ip availability
%{if private_network_only~}
# 10.0.0.1 is the gateway for private networks
- [ip, route, add, default, via, '10.0.0.1', dev, 'eth0']
%{else~}
# 172.31.1.1 is the gateway for public networks
- [ip, route, add, default, via, '172.31.1.1', dev, 'eth0']
%{endif~}


# Start the install-k3s-agent service
- ['/bin/bash', '/var/pre_install/install-k3s-agent.sh']
8 changes: 8 additions & 0 deletions templates/autoscaler.yaml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,14 @@ spec:
value: '${ipv4_subnet_id}'
- name: HCLOUD_FIREWALL
value: '${firewall_id}'
%{~ if disable_ipv4 ~}
- name: HCLOUD_PUBLIC_IPV4
value: "false"
%{~ endif ~}
%{~ if disable_ipv6 ~}
- name: HCLOUD_PUBLIC_IPV6
value: "false"
%{~ endif ~}
%{~ if cluster_autoscaler_server_creation_timeout != "" ~}
- name: HCLOUD_SERVER_CREATION_TIMEOUT
value: '${cluster_autoscaler_server_creation_timeout}'
Expand Down
13 changes: 13 additions & 0 deletions variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,19 @@ variable "autoscaler_taints" {
default = []
}

variable "autoscaler_disable_ipv4" {
description = "Disable IPv4 on nodes created by the Cluster Autoscaler."
type = bool
default = false
}

variable "autoscaler_disable_ipv6" {
description = "Disable IPv6 on nodes created by the Cluster Autoscaler."
type = bool
default = false
}


variable "hetzner_ccm_version" {
type = string
default = null
Expand Down
2 changes: 1 addition & 1 deletion versions.tf
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ terraform {
}
hcloud = {
source = "hetznercloud/hcloud"
version = ">= 1.49.1"
version = ">= 1.51.0"
}
local = {
source = "hashicorp/local"
Expand Down