# Complete Workflow Example: Kubernetes Cluster with Package System\n\nThis example demonstrates the complete workflow using the new KCL package and module loader system to deploy a production Kubernetes cluster.\n\n## Scenario\n\nDeploy a 3-node Kubernetes cluster with:\n\n- 1 master node\n- 2 worker nodes\n- Cilium CNI\n- Containerd runtime\n- UpCloud provider\n- Production-ready configuration\n\n## Prerequisites\n\n1. Core provisioning package installed\n2. UpCloud credentials configured\n3. SSH keys set up\n\n## Step 1: Environment Setup\n\n```\n# Ensure core package is installed\ncd /Users/Akasha/project-provisioning\n./provisioning/tools/kcl-packager.nu build --version 1.0.0\n./provisioning/tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz\n\n# Verify installation\nkcl list packages | grep provisioning\n```\n\n## Step 2: Create Workspace\n\n```\n# Create new workspace from template\nmkdir -p workspace/infra/production-k8s\ncd workspace/infra/production-k8s\n\n# Initialize workspace structure\n../../../provisioning/tools/workspace-init.nu . init\n\n# Verify structure\ntree -a .\n```\n\nExpected output:\n\n```\n.\n├── kcl.mod\n├── servers.k\n├── README.md\n├── .gitignore\n├── .taskservs/\n├── .providers/\n├── .clusters/\n├── .manifest/\n├── data/\n├── tmp/\n├── resources/\n└── clusters/\n```\n\n## Step 3: Discover Available Modules\n\n```\n# Discover available taskservs\n../../../provisioning/core/cli/module-loader discover taskservs\n\n# Search for Kubernetes-related modules\n../../../provisioning/core/cli/module-loader discover taskservs kubernetes\n\n# Discover providers\n../../../provisioning/core/cli/module-loader discover providers\n\n# Check output formats\n../../../provisioning/core/cli/module-loader discover taskservs --format json\n```\n\n## Step 4: Load Required Modules\n\n```\n# Load Kubernetes stack taskservs\n../../../provisioning/core/cli/module-loader load taskservs . [kubernetes, cilium, containerd]\n\n# Load UpCloud provider\n../../../provisioning/core/cli/module-loader load providers . [upcloud]\n\n# Verify loading\n../../../provisioning/core/cli/module-loader list taskservs .\n../../../provisioning/core/cli/module-loader list providers .\n```\n\nCheck generated files:\n\n```\n# Check auto-generated imports\ncat taskservs.k\ncat providers.k\n\n# Check manifest\ncat .manifest/taskservs.yaml\ncat .manifest/providers.yaml\n```\n\n## Step 5: Configure Infrastructure\n\nEdit `servers.k` to configure the Kubernetes cluster:\n\n```\n# Production Kubernetes Cluster Configuration\nimport provisioning.settings as settings\nimport provisioning.server as server\nimport provisioning.defaults as defaults\n\n# Import loaded modules (auto-generated)\nimport .taskservs.kubernetes.kubernetes as k8s\nimport .taskservs.cilium.cilium as cilium\nimport .taskservs.containerd.containerd as containerd\nimport .providers.upcloud as upcloud\n\n# Cluster settings\nk8s_settings: settings.Settings = {\n    main_name = "production-k8s"\n    main_title = "Production Kubernetes Cluster"\n\n    # Configure paths\n    settings_path = "./data/settings.yaml"\n    defaults_provs_dirpath = "./defs"\n    prov_data_dirpath = "./data"\n    created_taskservs_dirpath = "./tmp/k8s-deployment"\n    prov_resources_path = "./resources"\n    created_clusters_dirpath = "./tmp/k8s-clusters"\n    prov_clusters_path = "./clusters"\n\n    # Kubernetes cluster settings\n    cluster_admin_host = ""  # Set by provider (first master)\n    cluster_admin_port = 22\n    cluster_admin_user = "admin"\n    servers_wait_started = 60  # K8s nodes need more time\n\n    runset = {\n        wait = True\n        output_format = "human"\n        output_path = "tmp/k8s-deployment"\n        inventory_file = "./k8s-inventory.yaml"\n        use_time = True\n    }\n\n    # Secrets configuration\n    secrets = {\n        provider = "sops"\n        sops_config = {\n            age_key_file = "~/.age/keys.txt"\n            use_age = True\n        }\n    }\n}\n\n# Production Kubernetes cluster servers\nproduction_servers: [server.Server] = [\n    # Control plane node\n    {\n        hostname = "k8s-master-01"\n        title = "Kubernetes Master Node 01"\n\n        # Production specifications\n        time_zone = "UTC"\n        running_wait = 20\n        running_timeout = 400\n        storage_os_find = "name: debian-12 | arch: x86_64"\n\n        # Network configuration\n        network_utility_ipv4 = True\n        network_public_ipv4 = True\n        priv_cidr_block = "10.0.0.0/24"\n\n        # User settings\n        user = "admin"\n        user_ssh_port = 22\n        fix_local_hosts = True\n        labels = "env: production, role: control-plane, tier: master"\n\n        # Taskservs configuration\n        taskservs = [\n            {\n                name = "containerd"\n                profile = "production"\n                install_mode = "library"\n            },\n            {\n                name = "kubernetes"\n                profile = "master"\n                install_mode = "library-server"\n            },\n            {\n                name = "cilium"\n                profile = "master"\n                install_mode = "library"\n            }\n        ]\n    },\n\n    # Worker nodes\n    {\n        hostname = "k8s-worker-01"\n        title = "Kubernetes Worker Node 01"\n\n        time_zone = "UTC"\n        running_wait = 20\n        running_timeout = 400\n        storage_os_find = "name: debian-12 | arch: x86_64"\n\n        network_utility_ipv4 = True\n        network_public_ipv4 = True\n        priv_cidr_block = "10.0.0.0/24"\n\n        user = "admin"\n        user_ssh_port = 22\n        fix_local_hosts = True\n        labels = "env: production, role: worker, tier: compute"\n\n        taskservs = [\n            {\n                name = "containerd"\n                profile = "production"\n                install_mode = "library"\n            },\n            {\n                name = "kubernetes"\n                profile = "worker"\n                install_mode = "library"\n            },\n            {\n                name = "cilium"\n                profile = "worker"\n                install_mode = "library"\n            }\n        ]\n    },\n\n    {\n        hostname = "k8s-worker-02"\n        title = "Kubernetes Worker Node 02"\n\n        time_zone = "UTC"\n        running_wait = 20\n        running_timeout = 400\n        storage_os_find = "name: debian-12 | arch: x86_64"\n\n        network_utility_ipv4 = True\n        network_public_ipv4 = True\n        priv_cidr_block = "10.0.0.0/24"\n\n        user = "admin"\n        user_ssh_port = 22\n        fix_local_hosts = True\n        labels = "env: production, role: worker, tier: compute"\n\n        taskservs = [\n            {\n                name = "containerd"\n                profile = "production"\n                install_mode = "library"\n            },\n            {\n                name = "kubernetes"\n                profile = "worker"\n                install_mode = "library"\n            },\n            {\n                name = "cilium"\n                profile = "worker"\n                install_mode = "library"\n            }\n        ]\n    }\n]\n\n# Export for provisioning system\n{\n    settings = k8s_settings\n    servers = production_servers\n}\n```\n\n## Step 6: Validate Configuration\n\n```\n# Validate KCL configuration\nkcl run servers.k\n\n# Validate workspace\n../../../provisioning/core/cli/module-loader validate .\n\n# Check workspace info\n../../../provisioning/tools/workspace-init.nu . info\n```\n\n## Step 7: Configure Provider Credentials\n\n```\n# Create provider configuration directory\nmkdir -p defs\n\n# Create UpCloud provider defaults (example)\ncat > defs/upcloud_defaults.k << 'EOF'\n# UpCloud Provider Defaults\nimport provisioning.defaults as defaults\n\nupcloud_defaults: defaults.ServerDefaults = {\n    lock = False\n    time_zone = "UTC"\n    running_wait = 15\n    running_timeout = 300\n\n    # UpCloud specific settings\n    storage_os_find = "name: debian-12 | arch: x86_64"\n\n    # Network settings\n    network_utility_ipv4 = True\n    network_public_ipv4 = True\n\n    # SSH settings\n    ssh_key_path = "~/.ssh/id_rsa.pub"\n    user = "admin"\n    user_ssh_port = 22\n    fix_local_hosts = True\n\n    # UpCloud plan specifications\n    labels = "provider: upcloud"\n}\n\nupcloud_defaults\nEOF\n```\n\n## Step 8: Deploy Infrastructure\n\n```\n# Create servers with check mode first\n../../../provisioning/core/cli/provisioning server create --infra . --check\n\n# If validation passes, deploy for real\n../../../provisioning/core/cli/provisioning server create --infra .\n\n# Monitor server creation\n../../../provisioning/core/cli/provisioning server list --infra .\n```\n\n## Step 9: Install Taskservs\n\n```\n# Install containerd on all nodes\n../../../provisioning/core/cli/provisioning taskserv create containerd --infra .\n\n# Install Kubernetes (this will set up master and join workers)\n../../../provisioning/core/cli/provisioning taskserv create kubernetes --infra .\n\n# Install Cilium CNI\n../../../provisioning/core/cli/provisioning taskserv create cilium --infra .\n```\n\n## Step 10: Verify Cluster\n\n```\n# SSH to master node and verify cluster\n../../../provisioning/core/cli/provisioning server ssh k8s-master-01 --infra .\n\n# On the master node:\nkubectl get nodes\nkubectl get pods -A\nkubectl get services -A\n\n# Test Cilium connectivity\ncilium status\ncilium connectivity test\n```\n\n## Step 11: Deploy Sample Application\n\nCreate a test deployment to verify the cluster:\n\n```\n# Create namespace\nkubectl create namespace test-app\n\n# Deploy nginx\nkubectl create deployment nginx --image=nginx:latest -n test-app\nkubectl expose deployment nginx --port=80 --type=ClusterIP -n test-app\n\n# Verify deployment\nkubectl get pods -n test-app\nkubectl get services -n test-app\n```\n\n## Step 12: Cluster Management\n\n```\n# Add monitoring (example)\n../../../provisioning/core/cli/module-loader load taskservs . [prometheus, grafana]\n\n# Regenerate configuration\n../../../provisioning/core/cli/module-loader list taskservs .\n\n# Deploy monitoring stack\n../../../provisioning/core/cli/provisioning taskserv create prometheus --infra .\n../../../provisioning/core/cli/provisioning taskserv create grafana --infra .\n```\n\n## Step 13: Backup and Documentation\n\n```\n# Create cluster documentation\ncat > cluster-info.md << 'EOF'\n# Production Kubernetes Cluster\n\n## Cluster Details\n- **Name**: production-k8s\n- **Nodes**: 3 (1 master, 2 workers)\n- **CNI**: Cilium\n- **Runtime**: Containerd\n- **Provider**: UpCloud\n\n## Node Information\n- k8s-master-01: Control plane node\n- k8s-worker-01: Worker node\n- k8s-worker-02: Worker node\n\n## Loaded Modules\n- kubernetes (master/worker profiles)\n- cilium (cluster networking)\n- containerd (container runtime)\n- upcloud (cloud provider)\n\n## Management Commands\n```\n# SSH to master\n../../../provisioning/core/cli/provisioning server ssh k8s-master-01 --infra .\n\n# Update cluster\n../../../provisioning/core/cli/provisioning taskserv generate kubernetes --infra .\n```\n\nEOF\n\n# Backup workspace\n\ncp -r . ../production-k8s-backup-$(date +%Y%m%d)\n\n# Commit to version control\n\ngit add .\ngit commit -m "Initial Kubernetes cluster deployment with package system"\n\n```\n\n## Troubleshooting\n\n### Module Loading Issues\n```\n# If modules don't load properly\n../../../provisioning/core/cli/module-loader discover taskservs\n../../../provisioning/core/cli/module-loader load taskservs . [kubernetes, cilium, containerd] --force\n\n# Check generated imports\ncat taskservs.k\n```\n\n### KCL Compilation Issues\n\n```\n# Check for syntax errors\nkcl check servers.k\n\n# Validate specific schemas\nkcl run --dry-run servers.k\n```\n\n### Provider Authentication Issues\n\n```\n# Check provider configuration\ncat .providers/upcloud/provision_upcloud.k\n\n# Verify credentials\n../../../provisioning/core/cli/provisioning server price --provider upcloud\n```\n\n### Kubernetes Setup Issues\n\n```\n# Check taskserv logs\ntail -f tmp/k8s-deployment/kubernetes-*.log\n\n# Verify SSH connectivity\n../../../provisioning/core/cli/provisioning server ssh k8s-master-01 --infra . --command "systemctl status kubelet"\n```\n\n## Next Steps\n\n1. **Scale the cluster**: Add more worker nodes\n2. **Add storage**: Load and configure storage taskservs (rook-ceph, mayastor)\n3. **Setup monitoring**: Deploy Prometheus/Grafana stack\n4. **Configure ingress**: Set up ingress controllers\n5. **Implement GitOps**: Configure ArgoCD or Flux\n\nThis example demonstrates the complete workflow from workspace creation to production Kubernetes cluster deployment using the new package-based system.
