Provisioning a Private Talos Kubernetes Cluster on Hetzner Cloud

This is a follow up to Private Networking on Hetzner Cloud with Tailscale The previous post was about the network. This one is about what I put inside that network: a private Kubernetes cluster running Talos on Hetzner Cloud. The important part is not just “Kubernetes on Hetzner Cloud”. There are many posts about it. The part I cared about was making the cluster private from the first boot. No public IPs on the control plane. No public IPs on the workers. Access only through the Tailnet. That made Talos a good fit. No package manager, no SSH. You give it machine configuration, it becomes a Kubernetes node, and that is mostly it. Mostly. What I Wanted from the Cluster Private-only nodes : every Kubernetes node should live only on the Hetzner private network. Terraform-managed bootstrap : machines, Talos config, kubeconfig, and base add-ons should come from code. Talos : no manual server maintenance. Separate node pools : platform components should not fight application workloads. GitOps : Terraform can bootstrap ArgoCD, then ArgoCD owns the platform. The goal was to build something small enough that I could understand every moving part, but powerful enough that I could run actual projects on it. Cluster Shape The private network from the previous post gives the cluster a /24 to live in. I split that range into explicit chunks: Control plane: 10.0.128.16/28 Platform workers: 10.0.128.32/27 General workers: 10.0.128.64/27 Service network: 10.0.192.0/21 Pod network: 10.0.200.0/19 The control plane has three nodes. Platform workers run things like ArgoCD and platform components. General workers run applications like snapbyte.dev . flowchart TB Tailnet((Tailnet)) Internet((Internet)) subgraph VPC["Private network 10.0.0.0/16"] subgraph Subnet["Subnet 10.0.128.0/24"] NAT["NAT Gateway"] subgraph CP["Control Plane 10.0.128.16/28"] CP1["cp-1"] CP2["cp-2"] CP3["cp-3"] end subgraph Platform["Platform Workers 10.0.128.32/27"] ArgoCD["ArgoCD"] PlatformApps["Platform components"] end subgraph General["General Workers 10.0.128.64/27"] PublicApps["Public apps"] InternalApps["Internal apps"] end end end Tailnet -->|kubectl and talosctl| CP1 Tailnet --> Platform Tailnet --> General CP --> NAT Platform --> NAT General --> NAT NAT --> Internet The Kubernetes API endpoint is the first control plane node’s private IP: locals { cluster_endpoint = "https://${local.control_plane_private_ips[0]}:6443" } That endpoint is only useful if you are already inside the private network through Tailscale. Building the Talos Image Before Terraform could create any nodes, I needed a Talos image that Hetzner could boot. I started this cluster on Talos v1.11.3 . The later v1.12.6 upgrade came from an operational incident, not the initial design. Hetzner does not give you Talos as an image option, so I build my own snapshot with Packer. The flow is based on hcloud-talos/terraform-hcloud-talos . It starts a temporary Hetzner server, downloads the Talos raw image from the Talos Image Factory, writes it to disk, and saves the result as a snapshot. variable "talos_version" { type = string default = "v1.11.3" } source "hcloud" "talos" { rescue = "linux64" image = "debian-11" location = "nbg1" server_type = "cx22" ssh_username = "root" snapshot_name = "talos-${var.talos_version}-amd64" snapshot_labels = { type = "infra" os = "talos" version = var . talos_version arch = "amd64" } } The label part is the important bit. Terraform can later find the image by selector instead of relying on snapshot name: data "hcloud_image" "talos" { with_selector = "os=talos,type=infra,version=${var.talos_version},arch=amd64" } Worker Pools Before creating the machines, I needed a way to describe what kind of nodes I wanted. This is basically the same idea as node pools in managed Kubernetes offerings. GKE, EKS, and AKS all let you create groups of nodes with different sizes, labels, or taints. I wanted the same mental model. Each pool also gets its own Hetzner placement group. That tells Hetzner to spread the nodes in that pool across different physical hosts where possible. It does not make the pool highly available, but it avoids the failure mode where every platform worker ends up on the same machine. The pool config looks like this: worker_pools = { platform = { count = 3 sku = "cx33" cidr = "10.0.128.32/27" datacenter = "nbg1-dc3" labels = { purpose = "platform" } } general = { count = 3 sku = "cx23" cidr = "10.0.128.64/27" datacenter = "nbg1-dc3" labels = { purpose = "general" } } } This makes the Terraform code easier to reason about. It lets me create named groups of machines with known CIDR ranges, placement groups, and labels. Terraform Creates the Machines The node resources are just regular Hetzner servers, but with the public network disabled. resource "hcloud_server" "control_plane" { count = var . control_plane . count name = "${local.cluster_name}-cp-${count.index + 1}" datacenter = var . control_plane . datacenter image = data . hcloud_image . talos . id server_type = var . control_plane . sku public_net { ipv4_enabled = false ipv6_enabled = false } network { network_id = var . network_id ip = local . control_plane_ips [ count . index ] } } The worker pool map from the previous section gets flattened into individual servers. The code is not elegant, but the outcome is simple: if I add another worker to the general pool, it gets the next private IP in that pool and the right labels. locals { workers_flat = merge ([ for pool_name , pool_config in var . worker_pools : { for i in range ( 1 , pool_config . count + 1 ) : "${pool_name}-${i}" = > { pool = pool_name index = i sku = pool_config . sku datacenter = pool_config . datacenter labels = pool_config . labels } } ]...) } Talos Bootstraps Kubernetes Once the servers exist, the Talos Terraform provider takes over. It generates machine secrets, creates control plane and worker configs, applies patches, bootstraps the first control plane node, waits for Talos cluster health, and gives me a kubeconfig. There is one base config for control plane nodes, and one worker base config per pool: data "talos_machine_configuration" "control_plane" { cluster_name = local . cluster_name cluster_endpoint = local . cluster_endpoint machine_type = "controlplane" machine_secrets = talos_machine_secrets . this . machine_secrets talos_version = var . talos_version kubernetes_version = var . kubernetes_version } data "talos_machine_configuration" "worker" { for_each = var . worker_pools cluster_name = local . cluster_name cluster_endpoint = local . cluster_endpoint machine_type = "worker" machine_secrets = talos_machine_secrets . this . machine_secrets talos_version = var . talos_version kubernetes_version = var . kubernetes_version } Then Terraform applies the patched control-plane config to each control-plane node: resource "talos_machine_configuration_apply" "control_plane" { count = var . control_plane . count client_configuration = talos_machine_secrets . this . client_configuration machine_configuration_input = data . talos_machine_configuration . control_plane . machine_configuration node = local . control_plane_private_ips [ count . index ] config_patches = [ yamlencode ( local . control_plane_patch ), ] } Workers follow the same pattern, except the patch comes from the worker pool: resource "talos_machine_configuration_apply" "worker" { for_each = local . workers_flat client_configuration = talos_machine_secrets . this . client_configuration machine_configuration_input = data . talos_machine_configuration . worker [ each . value . pool ]. machine_configuration node = flatten ( hcloud_server . worker [ each . key ]. network )[ 0 ]. ip config_patches = [ yamlencode ( local . worker_pool_patches [ each . value . pool ]), ] } Nodes need the right installer image, node IP selection, default route, pod and service CIDRs, and CNI behavior. This is the part I messed up and later caused the first real failure. A simplified version of the patch, showing the final intent,…