provisioning-catalog/components/private_ingress

private_ingress

VPN-reachable L7 ingress for *.in.librecloud.online services. One private IP (10.200.3.6), host-based routing, HTTPS via cert-manager DNS-01.

Architecture

VPN client → 10.200.3.6 (L4 LB :80/:443)
               └─ sozu pod (ingress-private ns)
                    ├─ Host: grafana.in     → grafana.observability:3000
                    ├─ Host: prometheus.in  → prometheus.observability:9090
                    └─ Host: longhorn.in    → longhorn-frontend.longhorn-system:80

The L4 LoadBalancer (type=LoadBalancer, lbipam.cilium.io/ips: 10.200.3.6) is the only layer that crosses the WireGuard datapath from wrk-0. The Cilium GatewayAPI private-gateway (10.200.3.3) is NOT reachable from WG ingress (ADR-018 constraint).

Proxy engine: sozu

sozu was chosen over Caddy, nginx, HAProxy, and Envoy. Decision rationale:

• Rust-native: aligns with the project stack (ADR-007).
• Rustls crypto provider: TLS 1.3, modern cipher suites — no OpenSSL or Go crypto/tls dependency.
• Hot-reload via sozuctl: sozuctl certificate replace reloads a cert per cluster without dropping connections and without restarting the process. The control plane is a UNIX socket (/var/run/sozu/sozu.sock); the sidecar calls it on cert renewal. Caddy requires SIGHUP; nginx requires SIGHUP + graceful restart.
• Live operator TUI: sozuctl status / sozuctl metrics for runtime inspection without kubectl exec.

Rejected alternatives:

• Caddy: equal cert-file integration, no hot-reload equivalent to sozuctl, Go runtime.
• nginx: adequate but verbose (15 lines per route vs ~8 for sozu TOML); aralez already handles static-file serving, this component is pure reverse proxy.
• Envoy: Cilium already runs Envoy for GatewayAPI internally; a second standalone Envoy causes operational confusion. Static Envoy YAML is ~60-80 lines per virtualhost.
• HAProxy: requires PEM concatenation (tls.crt + tls.key) before startup — extra init container step not needed with sozu or nginx.

TLS rotation: Pattern B (cert-manager CSI driver + sozuctl sidecar)

Requires csi_driver_enabled = true in the cert_manager component.

1. cert-manager CSI driver DaemonSet runs on all nodes (installed via just update-cert-manager).
2. The private-ingress Deployment declares one CSI volume per route:

   volumes:
     - name: tls-<slug>
       csi:
         driver: csi.cert-manager.io
         readOnly: true
         volumeAttributes:
           csi.cert-manager.io/issuer-name: letsencrypt-prod-librecloud
           csi.cert-manager.io/issuer-kind: ClusterIssuer
           csi.cert-manager.io/dns-names: "grafana.in.librecloud.online"
   

3. The kubelet calls the CSI driver node plugin to mount the cert as tmpfs at /etc/sozu/tls/<host>/tls.crt + tls.key. No Certificate CR or Secret is created — the cert lifecycle is ephemeral and pod-scoped.
4. The sozuctl-sidecar (inotifywait loop) detects cert file writes on renewal and calls sozuctl certificate replace per route — sozu hot-reloads without pod restart.

No Certificate CRs or Secret mounts are needed in the manifest plan. The cluster_issuer field in the NCL contract selects which ClusterIssuer the CSI driver references. For libre-wuji: letsencrypt-prod-librecloud (DNS-01 via Cloudflare).

tls_rotation field

Value	Behaviour
'csi_driver	Pattern B — CSI driver + sozuctl sidecar. Mandatory for libre-wuji.
'reloader	cert-manager Secret mount + stakater/Reloader rollout on rotation.
'sozu_managed	sozu's built-in ACME client (no cert-manager dependency).

routes NCL schema

routes = [
  { host = "grafana.in.librecloud.online",   upstream = "grafana.observability.svc.libre-wuji.local",             port = 3000 },
  { host = "prometheus.in.librecloud.online", upstream = "prometheus.observability.svc.libre-wuji.local",          port = 9090 },
  { host = "longhorn.in.librecloud.online",  upstream = "longhorn-frontend.longhorn-system.svc.libre-wuji.local", port = 80   },
]

Adding a new .in service: append one { host, upstream, port } entry to routes + one dns_internal make_route line in the workspace NCL + just install-private-ingress update.

DNS note

dns_internal entries use zone in.librecloud.online (NOT librecloud.online). The split-horizon zone in coredns_vpn_split.ncl is keyed by the SOA zone name. Using the wrong zone silently mis-files the record and WG clients cannot resolve the host.

Backend DNS caveat

sozu [[clusters.<name>.backends]] ip_address is resolved at startup only. K8s Service ClusterIPs are stable, so hostname-based backends (FQDN via cluster DNS) are safe in practice. Verify that the pinned sozu version accepts hostnames — if not, resolve ClusterIP at deploy time in vars.nu via kubectl get svc.