This site used to run on Docker Compose behind Traefik and nginx. I wanted to try something closer to how a real cluster is run, so I tore it down and put k3s directly on the box instead — one lightweight Kubernetes node hosting this site along with a couple of others. It went about as smoothly as homelab infrastructure ever does, which is to say: not smoothly, but instructively.
SELinux wants a say first
Rocky Linux ships with SELinux enforcing by default, and the k3s installer doesn't pull in what it needs on its own. Installing container-selinux and selinux-policy-base before running the install script saved a round of confusing failures later.
A Traefik crash that fixed itself
Right after install, the bundled Traefik job went into CrashLoopBackOff complaining that its CRDs were missing. Classic chicken-and-egg: a separate job installs the CRDs, and Traefik's own install job was racing it. No fix needed — it retried and passed once the CRD job caught up.
502s that made no sense
Every request came back 502, even plain requests to the homepage. The confusing part: curling a backend pod's IP directly from the node worked fine, which pointed away from the real problem. The actual issue was that firewalld was blocking pod-to-pod traffic — host-to-pod worked because it doesn't cross that boundary, but Traefik proxying to a backend pod does. Trusting the pod and service CIDRs in firewalld's trusted zone fixed it immediately.
Reloading firewalld broke the exit path too
Fixing the trusted zone required a firewall-cmd --reload, which turned out to knock over k3s's own iptables rules for outbound/NodePort traffic. A systemctl restart k3s made it reconcile its rules again — worth remembering any time firewalld gets reloaded on a k3s box.
The last mile wasn't mine to fix
Even with everything above sorted, external requests still timed out. Nothing left on the box explained it — which meant it wasn't the box. The hosting provider's own cloud firewall was blocking inbound 80/443, most likely reset during a prior reprovision. That one just needed opening from the provider's control panel, not SSH.
cert-manager holds a grudge
Once traffic actually reached the cluster, Let's Encrypt certificates still refused to issue. Deleting the failed CertificateRequest did nothing — it turns out cert-manager tracks issuance backoff on the Certificate resource itself, not the request. Deleting the Certificate was the fix: since it's owned by the Ingress, it was recreated instantly with a clean history, and it issued within a minute.
Four bugs, three of them wearing a firewall costume
None of this was exotic — SELinux, two layers of firewalld, a cloud provider's own edge, and cert-manager's retry logic. But each one hid the next one behind it, so the fix only looked obvious in hindsight. All three domains are now serving real TLS certificates off a single k3s node, which is exactly the point of running a homelab: nothing here was necessary, and now I understand all of it a little better.