Major bug: ssh inaccessible with any port after sed -i 's/^#*Port .*/Port {{ ssh_port }}/' /etc/ssh/sshd_config

[tgrushka]

A change seems to have been made recently that renders SSH completely inaccessible (in failed state) after it changes the sshd_config to use {{ ssh_port }} as the port, which is not valid.

This file: https://github.com/vitobotta/hetzner-k3s/blob/main/templates/ssh/configure_ssh.sh#L9

The only way to login is via Rescue, reset root password, login with console.

Config:

hetzner_token: "my_hetzner_token_blah"
cluster_name: k3s
k3s_version: v1.33.1+k3s1
kubeconfig_path: "./kubeconfig.yaml"

networking:
  ssh:
    port: 22
    private_key_path: ./id_ed25519
    public_key_path: ./id_ed25519.pub
  allowed_networks:
    ssh:
      - "100.64.0.0/10" # Tailnet
      - "10.3.0.0/16" # Hetzner private network
      - "blah.blah.blah.blah/32" # Current IP address (replaced in `just create`)
    api:
      - "100.64.0.0/10" # Tailnet
      - "10.3.0.0/16" # Hetzner private network
      - "blah.blah.blah.blah/32" # Current IP address (replaced in `just create`)
  public_network:
    ipv4: true
    ipv6: true
  private_network:
    enabled: true
    subnet: "10.3.0.0/16" # Hetzner private network
    existing_network_name: "k3s-net"
  cni:
    enabled: true
    encryption: false
    mode: flannel

datastore:
  mode: etcd

image: debian-12

# Taint masters, especially when using smaller cpx21
schedule_workloads_on_masters: false

masters_pool:
  instance_type: cpx21 # 3 vCPU, 4 GB RAM, 80 GB SSD
  instance_count: 1
  locations: [hil]

worker_node_pools:
- name: cpx21
  instance_type: cpx21
  location: hil
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 5

embedded_registry_mirror:
  # "Caching" -- not needed for small clusters or < large container image sizes.
  enabled: false

protect_against_deletion: false

# Not needed for 1 control node; we will also use headscale / tailscale to access.
create_load_balancer_for_the_kubernetes_api: false

# how many nodes to upgrade at the same time
k3s_upgrade_concurrency: 1

# Install tailscale (ssh still fails when commented out)
# post_create_commands:
# - apt update && apt upgrade -y && apt autoremove -y
# - curl -fsSL https://pkgs.tailscale.com/stable/debian/bookworm.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
# - curl -fsSL https://pkgs.tailscale.com/stable/debian/bookworm.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list
# - apt update && apt install -y tailscale
# - tailscale up --reset --login-server="http://10.3.0.2:8080" --authkey="blahblahblahblah"

# kube_api_server_args:
# - arg1
# kube_scheduler_args:
# - arg1
# kube_controller_manager_args:
# - arg1
# kube_cloud_controller_manager_args:
# - arg1
# kubelet_args:
# - arg1
# kube_proxy_args:
# - arg1

Log:

�[32m _          _                            _    _____     �[0m
�[32m| |__   ___| |_ _____ __   ___ _ __     | | _|___ / ___ �[0m
�[32m| '_ \ / _ \ __|_  / '_ \ / _ \ '__|____| |/ / |_ \/ __|�[0m
�[32m| | | |  __/ |_ / /| | | |  __/ | |_____|   < ___) \__ \�[0m
�[32m|_| |_|\___|\__/___|_| |_|\___|_|       |_|\_\____/|___/�[0m

�[34mVersion: 2.3.0�[0m

[Configuration] Validating configuration...
[Configuration] ...configuration seems valid.
[Placement groups] Creating placement group k3s-masters...
[Placement groups] ...placement group k3s-masters created
[Instance k3s-master1] Creating instance k3s-master1 (attempt 1)...
[Instance k3s-master1] Instance status: starting
[Instance k3s-master1] Powering on instance (attempt 1)
[Instance k3s-master1] Waiting for instance to be powered on...
[Instance k3s-master1] Instance status: running
[Instance k3s-master1] SSH command failed: debug1: Reading configuration data /Users/tom/.ssh/config
debug1: /Users/tom/.ssh/config line 4: Applying options for *
debug1: /Users/tom/.ssh/config line 8: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/100-macos.conf
debug1: /etc/ssh/ssh_config.d/100-macos.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/crypto.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to INSTANCE_PUBLIC_IP [INSTANCE_PUBLIC_IP] port 22.
debug1: connect to address INSTANCE_PUBLIC_IP port 22: Connection refused
ssh: connect to host INSTANCE_PUBLIC_IP port 22: Connection refused
SSH command result: ======
SSH command expected: ===ready===
Matching?: ===false===
[Instance k3s-master1] Waiting for instance k3s-master1 to be ready...
[Instance k3s-master1] ready
SSH command result: ===ready===
SSH command expected: ===ready===
Matching?: ===true===
[Instance k3s-master1] ...instance k3s-master1 created
[Firewall] Updating firewall...
[Firewall] ...firewall updated
[Instance k3s-master1] SSH command failed: debug1: Reading configuration data /Users/tom/.ssh/config
debug1: /Users/tom/.ssh/config line 4: Applying options for *
debug1: /Users/tom/.ssh/config line 8: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/100-macos.conf
debug1: /etc/ssh/ssh_config.d/100-macos.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/crypto.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to INSTANCE_PUBLIC_IP [INSTANCE_PUBLIC_IP] port 22.
debug1: connect to address INSTANCE_PUBLIC_IP port 22: Connection refused
ssh: connect to host INSTANCE_PUBLIC_IP port 22: Connection refused
[Instance k3s-master1] SSH command failed: debug1: Reading configuration data /Users/tom/.ssh/config
debug1: /Users/tom/.ssh/config line 4: Applying options for *
debug1: /Users/tom/.ssh/config line 8: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/100-macos.conf
debug1: /etc/ssh/ssh_config.d/100-macos.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/crypto.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to INSTANCE_PUBLIC_IP [INSTANCE_PUBLIC_IP] port 22.
debug1: connect to address INSTANCE_PUBLIC_IP port 22: Connection refused
ssh: connect to host INSTANCE_PUBLIC_IP port 22: Connection refused
[Instance k3s-master1] SSH command failed: debug1: Reading configuration data /Users/tom/.ssh/config
debug1: /Users/tom/.ssh/config line 4: Applying options for *
debug1: /Users/tom/.ssh/config line 8: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/100-macos.conf
debug1: /etc/ssh/ssh_config.d/100-macos.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/crypto.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to INSTANCE_PUBLIC_IP [INSTANCE_PUBLIC_IP] port 22.
debug1: connect to address INSTANCE_PUBLIC_IP port 22: Connection refused
ssh: connect to host INSTANCE_PUBLIC_IP port 22: Connection refused
[Instance k3s-master1] SSH command failed: debug1: Reading configuration data /Users/tom/.ssh/config
debug1: /Users/tom/.ssh/config line 4: Applying options for *
debug1: /Users/tom/.ssh/config line 8: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/100-macos.conf
debug1: /etc/ssh/ssh_config.d/100-macos.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/crypto.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to INSTANCE_PUBLIC_IP [INSTANCE_PUBLIC_IP] port 22.
debug1: connect to address INSTANCE_PUBLIC_IP port 22: Connection refused
ssh: connect to host INSTANCE_PUBLIC_IP port 22: Connection refused
[Instance k3s-master1] Waiting for the control plane to be ready...
[Control plane] Generating the kubeconfig file to /Users/tom/apps/k3s-hetzner/kubeconfig.yaml...
[Instance k3s-master1] SSH command failed: debug1: Reading configuration data /Users/tom/.ssh/config
debug1: /Users/tom/.ssh/config line 4: Applying options for *
debug1: /Users/tom/.ssh/config line 8: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/100-macos.conf
debug1: /etc/ssh/ssh_config.d/100-macos.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/crypto.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to INSTANCE_PUBLIC_IP [INSTANCE_PUBLIC_IP] port 22.
debug1: connect to address INSTANCE_PUBLIC_IP port 22: Connection refused
ssh: connect to host INSTANCE_PUBLIC_IP port 22: Connection refused
[Control plane] : error: no context exists with the name: "k3s-master1"

[tgrushka]

Found an automated fix:

post_create_commands:
# The Med File Unf*cker https://youtu.be/Y9clBHENy4Q?t=156
# Remove when fixed: https://github.com/vitobotta/hetzner-k3s/blob/main/templates/ssh/configure_ssh.sh#L10
- sed -i 's/{{ ssh_port }}/22/' /etc/ssh/sshd_config
- systemctl restart ssh
- "echo 'sshd_config unf*cked successfully 🤣'"
# Now install tailscale or do other stuff...

Now, I preprocess my config through 1Password to fill in my auth keys, which requires further escaping the double brackets:

- sed -i 's/{{ "{{" }} ssh_port {{ "}}" }}/22/' /etc/ssh/sshd_config

But how to get it to replace it with networking.ssh.port without changing the scripts, I don't know.

[vitobotta]

Hi, this has been reported in a discussion too and I am fixing it in #599. You shouldn't need to use post create commands to fix this. It should be handled by hetzner-k3s directly. Can you try v2.3.1.rc1 once it's ready?

[vitobotta]

This is fixed in v2.3.1.