102 lines
5 KiB
Markdown
102 lines
5 KiB
Markdown
# private_ingress
|
|
|
|
VPN-reachable L7 ingress for `*.in.librecloud.online` services. One private IP
|
|
(`10.200.3.6`), host-based routing, HTTPS via cert-manager DNS-01.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
VPN client → 10.200.3.6 (L4 LB :80/:443)
|
|
└─ sozu pod (ingress-private ns)
|
|
├─ Host: grafana.in → grafana.observability:3000
|
|
├─ Host: prometheus.in → prometheus.observability:9090
|
|
└─ Host: longhorn.in → longhorn-frontend.longhorn-system:80
|
|
```
|
|
|
|
The L4 LoadBalancer (`type=LoadBalancer`, `lbipam.cilium.io/ips: 10.200.3.6`) is the
|
|
only layer that crosses the WireGuard datapath from wrk-0. The Cilium GatewayAPI
|
|
private-gateway (10.200.3.3) is NOT reachable from WG ingress (ADR-018 constraint).
|
|
|
|
## Proxy engine: sozu
|
|
|
|
sozu was chosen over Caddy, nginx, HAProxy, and Envoy. Decision rationale:
|
|
|
|
- **Rust-native**: aligns with the project stack (ADR-007).
|
|
- **Rustls crypto provider**: TLS 1.3, modern cipher suites — no OpenSSL or Go
|
|
crypto/tls dependency.
|
|
- **Hot-reload via sozuctl**: `sozuctl certificate replace` reloads a cert per cluster
|
|
without dropping connections and without restarting the process. The control plane
|
|
is a UNIX socket (`/var/run/sozu/sozu.sock`); the sidecar calls it on cert renewal.
|
|
Caddy requires SIGHUP; nginx requires SIGHUP + graceful restart.
|
|
- **Live operator TUI**: `sozuctl status` / `sozuctl metrics` for runtime inspection
|
|
without `kubectl exec`.
|
|
|
|
Rejected alternatives:
|
|
- **Caddy**: equal cert-file integration, no hot-reload equivalent to sozuctl, Go runtime.
|
|
- **nginx**: adequate but verbose (15 lines per route vs ~8 for sozu TOML); aralez already
|
|
handles static-file serving, this component is pure reverse proxy.
|
|
- **Envoy**: Cilium already runs Envoy for GatewayAPI internally; a second standalone Envoy
|
|
causes operational confusion. Static Envoy YAML is ~60-80 lines per virtualhost.
|
|
- **HAProxy**: requires PEM concatenation (tls.crt + tls.key) before startup — extra init
|
|
container step not needed with sozu or nginx.
|
|
|
|
## TLS rotation: Pattern B (cert-manager CSI driver + sozuctl sidecar)
|
|
|
|
Requires `csi_driver_enabled = true` in the `cert_manager` component.
|
|
|
|
1. cert-manager CSI driver DaemonSet runs on all nodes (installed via `just update-cert-manager`).
|
|
2. The private-ingress Deployment declares one CSI volume per route:
|
|
```yaml
|
|
volumes:
|
|
- name: tls-<slug>
|
|
csi:
|
|
driver: csi.cert-manager.io
|
|
readOnly: true
|
|
volumeAttributes:
|
|
csi.cert-manager.io/issuer-name: letsencrypt-prod-librecloud
|
|
csi.cert-manager.io/issuer-kind: ClusterIssuer
|
|
csi.cert-manager.io/dns-names: "grafana.in.librecloud.online"
|
|
```
|
|
3. The kubelet calls the CSI driver node plugin to mount the cert as `tmpfs` at
|
|
`/etc/sozu/tls/<host>/tls.crt` + `tls.key`. No `Certificate` CR or `Secret` is
|
|
created — the cert lifecycle is ephemeral and pod-scoped.
|
|
4. The sozuctl-sidecar (`inotifywait` loop) detects cert file writes on renewal and
|
|
calls `sozuctl certificate replace` per route — sozu hot-reloads without pod restart.
|
|
|
|
No `Certificate` CRs or `Secret` mounts are needed in the manifest plan.
|
|
The `cluster_issuer` field in the NCL contract selects which ClusterIssuer the CSI
|
|
driver references. For libre-wuji: `letsencrypt-prod-librecloud` (DNS-01 via Cloudflare).
|
|
|
|
## `tls_rotation` field
|
|
|
|
| Value | Behaviour |
|
|
|---------------|-----------------------------------------------------------------------|
|
|
| `'csi_driver` | Pattern B — CSI driver + sozuctl sidecar. Mandatory for libre-wuji. |
|
|
| `'reloader` | cert-manager Secret mount + stakater/Reloader rollout on rotation. |
|
|
| `'sozu_managed` | sozu's built-in ACME client (no cert-manager dependency). |
|
|
|
|
## `routes` NCL schema
|
|
|
|
```nickel
|
|
routes = [
|
|
{ host = "grafana.in.librecloud.online", upstream = "grafana.observability.svc.libre-wuji.local", port = 3000 },
|
|
{ host = "prometheus.in.librecloud.online", upstream = "prometheus.observability.svc.libre-wuji.local", port = 9090 },
|
|
{ host = "longhorn.in.librecloud.online", upstream = "longhorn-frontend.longhorn-system.svc.libre-wuji.local", port = 80 },
|
|
]
|
|
```
|
|
|
|
Adding a new `.in` service: append one `{ host, upstream, port }` entry to `routes` +
|
|
one `dns_internal` `make_route` line in the workspace NCL + `just install-private-ingress update`.
|
|
|
|
## DNS note
|
|
|
|
`dns_internal` entries use zone `in.librecloud.online` (NOT `librecloud.online`). The
|
|
split-horizon zone in `coredns_vpn_split.ncl` is keyed by the SOA zone name. Using the
|
|
wrong zone silently mis-files the record and WG clients cannot resolve the host.
|
|
|
|
## Backend DNS caveat
|
|
|
|
sozu `[[clusters.<name>.backends]]` `ip_address` is resolved at startup only. K8s
|
|
Service ClusterIPs are stable, so hostname-based backends (FQDN via cluster DNS) are
|
|
safe in practice. Verify that the pinned sozu version accepts hostnames — if not,
|
|
resolve ClusterIP at deploy time in `vars.nu` via `kubectl get svc`.
|