# Infrastructure Management Guide\n\nThis comprehensive guide covers creating, managing, and maintaining infrastructure using Infrastructure Automation.\n\n## What You'll Learn\n\n- Infrastructure lifecycle management\n- Server provisioning and management\n- Task service installation and configuration\n- Cluster deployment and orchestration\n- Scaling and optimization strategies\n- Monitoring and maintenance procedures\n- Cost management and optimization\n\n## Infrastructure Concepts\n\n### Infrastructure Components\n\n| Component | Description | Examples |\n| ----------- | ------------- | ---------- |\n| **Servers** | Virtual machines or containers | Web servers, databases, workers |\n| **Task Services** | Software installed on servers | Kubernetes, Docker, databases |\n| **Clusters** | Groups of related services | Web clusters, database clusters |\n| **Networks** | Connectivity between resources | VPCs, subnets, load balancers |\n| **Storage** | Persistent data storage | Block storage, object storage |\n\n### Infrastructure Lifecycle\n\n```\nPlan → Create → Deploy → Monitor → Scale → Update → Retire\n```\n\nEach phase has specific commands and considerations.\n\n## Server Management\n\n### Understanding Server Configuration\n\nServers are defined in Nickel configuration files:\n\n```\n# Example server configuration\nimport models.server\n\nservers: [\n    server.Server {\n        name = "web-01"\n        provider = "aws"          # aws, upcloud, local\n        plan = "t3.medium"        # Instance type/plan\n        os = "ubuntu-22.04"       # Operating system\n        zone = "us-west-2a"       # Availability zone\n\n        # Network configuration\n        vpc = "main"\n        subnet = "web"\n        security_groups = ["web", "ssh"]\n\n        # Storage configuration\n        storage = {\n            root_size = "50 GB"\n            additional = [\n                {name = "data", size = "100 GB", type = "gp3"}\n            ]\n        }\n\n        # Task services to install\n        taskservs = [\n            "containerd",\n            "kubernetes",\n            "monitoring"\n        ]\n\n        # Tags for organization\n        tags = {\n            environment = "production"\n            team = "platform"\n            cost_center = "engineering"\n        }\n    }\n]\n```\n\n### Server Lifecycle Commands\n\n#### Creating Servers\n\n```\n# Plan server creation (dry run)\nprovisioning server create --infra my-infra --check\n\n# Create servers\nprovisioning server create --infra my-infra\n\n# Create with specific parameters\nprovisioning server create --infra my-infra --wait --yes\n\n# Create single server type\nprovisioning server create web --infra my-infra\n```\n\n#### Managing Existing Servers\n\n```\n# List all servers\nprovisioning server list --infra my-infra\n\n# Show detailed server information\nprovisioning show servers --infra my-infra\n\n# Show specific server\nprovisioning show servers web-01 --infra my-infra\n\n# Get server status\nprovisioning server status web-01 --infra my-infra\n```\n\n#### Server Operations\n\n```\n# Start/stop servers\nprovisioning server start web-01 --infra my-infra\nprovisioning server stop web-01 --infra my-infra\n\n# Restart servers\nprovisioning server restart web-01 --infra my-infra\n\n# Resize server\nprovisioning server resize web-01 --plan t3.large --infra my-infra\n\n# Update server configuration\nprovisioning server update web-01 --infra my-infra\n```\n\n#### SSH Access\n\n```\n# SSH to server\nprovisioning server ssh web-01 --infra my-infra\n\n# SSH with specific user\nprovisioning server ssh web-01 --user admin --infra my-infra\n\n# Execute command on server\nprovisioning server exec web-01 "systemctl status kubernetes" --infra my-infra\n\n# Copy files to/from server\nprovisioning server copy local-file.txt web-01:/tmp/ --infra my-infra\nprovisioning server copy web-01:/var/log/app.log ./logs/ --infra my-infra\n```\n\n#### Server Deletion\n\n```\n# Plan server deletion (dry run)\nprovisioning server delete --infra my-infra --check\n\n# Delete specific server\nprovisioning server delete web-01 --infra my-infra\n\n# Delete with confirmation\nprovisioning server delete web-01 --infra my-infra --yes\n\n# Delete but keep storage\nprovisioning server delete web-01 --infra my-infra --keepstorage\n```\n\n## Task Service Management\n\n### Understanding Task Services\n\nTask services are software components installed on servers:\n\n- **Container Runtimes**: containerd, cri-o, docker\n- **Orchestration**: kubernetes, nomad\n- **Networking**: cilium, calico, haproxy\n- **Storage**: rook-ceph, longhorn, nfs\n- **Databases**: postgresql, mysql, mongodb\n- **Monitoring**: prometheus, grafana, alertmanager\n\n### Task Service Configuration\n\n```\n# Task service configuration example\ntaskservs: {\n    kubernetes: {\n        version = "1.28"\n        network_plugin = "cilium"\n        ingress_controller = "nginx"\n        storage_class = "gp3"\n\n        # Cluster configuration\n        cluster = {\n            name = "production"\n            pod_cidr = "10.244.0.0/16"\n            service_cidr = "10.96.0.0/12"\n        }\n\n        # Node configuration\n        nodes = {\n            control_plane = ["master-01", "master-02", "master-03"]\n            workers = ["worker-01", "worker-02", "worker-03"]\n        }\n    }\n\n    postgresql: {\n        version = "15"\n        port = 5432\n        max_connections = 200\n        shared_buffers = "256 MB"\n\n        # High availability\n        replication = {\n            enabled = true\n            replicas = 2\n            sync_mode = "synchronous"\n        }\n\n        # Backup configuration\n        backup = {\n            enabled = true\n            schedule = "0 2 * * *"  # Daily at 2 AM\n            retention = "30d"\n        }\n    }\n}\n```\n\n### Task Service Commands\n\n#### Installing Services\n\n```\n# Install single service\nprovisioning taskserv create kubernetes --infra my-infra\n\n# Install multiple services\nprovisioning taskserv create containerd kubernetes cilium --infra my-infra\n\n# Install with specific version\nprovisioning taskserv create kubernetes --version 1.28 --infra my-infra\n\n# Install on specific servers\nprovisioning taskserv create postgresql --servers db-01,db-02 --infra my-infra\n```\n\n#### Managing Services\n\n```\n# List available services\nprovisioning taskserv list\n\n# List installed services\nprovisioning taskserv list --infra my-infra --installed\n\n# Show service details\nprovisioning taskserv show kubernetes --infra my-infra\n\n# Check service status\nprovisioning taskserv status kubernetes --infra my-infra\n\n# Check service health\nprovisioning taskserv health kubernetes --infra my-infra\n```\n\n#### Service Operations\n\n```\n# Start/stop services\nprovisioning taskserv start kubernetes --infra my-infra\nprovisioning taskserv stop kubernetes --infra my-infra\n\n# Restart services\nprovisioning taskserv restart kubernetes --infra my-infra\n\n# Update services\nprovisioning taskserv update kubernetes --infra my-infra\n\n# Configure services\nprovisioning taskserv configure kubernetes --config cluster.yaml --infra my-infra\n```\n\n#### Service Removal\n\n```\n# Remove service\nprovisioning taskserv delete kubernetes --infra my-infra\n\n# Remove with data cleanup\nprovisioning taskserv delete postgresql --cleanup-data --infra my-infra\n\n# Remove from specific servers\nprovisioning taskserv delete kubernetes --servers worker-03 --infra my-infra\n```\n\n### Version Management\n\n```\n# Check for updates\nprovisioning taskserv check-updates --infra my-infra\n\n# Check specific service updates\nprovisioning taskserv check-updates kubernetes --infra my-infra\n\n# Show available versions\nprovisioning taskserv versions kubernetes\n\n# Upgrade to latest version\nprovisioning taskserv upgrade kubernetes --infra my-infra\n\n# Upgrade to specific version\nprovisioning taskserv upgrade kubernetes --version 1.29 --infra my-infra\n```\n\n## Cluster Management\n\n### Understanding Clusters\n\nClusters are collections of services that work together to provide functionality:\n\n```\n# Cluster configuration example\nclusters: {\n    web_cluster: {\n        name = "web-application"\n        description = "Web application cluster"\n\n        # Services in the cluster\n        services = [\n            {\n                name = "nginx"\n                replicas = 3\n                image = "nginx:1.24"\n                ports = [80, 443]\n            }\n            {\n                name = "app"\n                replicas = 5\n                image = "myapp:latest"\n                ports = [8080]\n            }\n        ]\n\n        # Load balancer configuration\n        load_balancer = {\n            type = "application"\n            health_check = "/health"\n            ssl_cert = "wildcard.example.com"\n        }\n\n        # Auto-scaling\n        auto_scaling = {\n            min_replicas = 2\n            max_replicas = 10\n            target_cpu = 70\n            target_memory = 80\n        }\n    }\n}\n```\n\n### Cluster Commands\n\n#### Creating Clusters\n\n```\n# Create cluster\nprovisioning cluster create web-cluster --infra my-infra\n\n# Create with specific configuration\nprovisioning cluster create web-cluster --config cluster.yaml --infra my-infra\n\n# Create and deploy\nprovisioning cluster create web-cluster --deploy --infra my-infra\n```\n\n#### Managing Clusters\n\n```\n# List available clusters\nprovisioning cluster list\n\n# List deployed clusters\nprovisioning cluster list --infra my-infra --deployed\n\n# Show cluster details\nprovisioning cluster show web-cluster --infra my-infra\n\n# Get cluster status\nprovisioning cluster status web-cluster --infra my-infra\n```\n\n#### Cluster Operations\n\n```\n# Deploy cluster\nprovisioning cluster deploy web-cluster --infra my-infra\n\n# Scale cluster\nprovisioning cluster scale web-cluster --replicas 10 --infra my-infra\n\n# Update cluster\nprovisioning cluster update web-cluster --infra my-infra\n\n# Rolling update\nprovisioning cluster update web-cluster --rolling --infra my-infra\n```\n\n#### Cluster Deletion\n\n```\n# Delete cluster\nprovisioning cluster delete web-cluster --infra my-infra\n\n# Delete with data cleanup\nprovisioning cluster delete web-cluster --cleanup --infra my-infra\n```\n\n## Network Management\n\n### Network Configuration\n\n```\n# Network configuration\nnetwork: {\n    vpc = {\n        cidr = "10.0.0.0/16"\n        enable_dns = true\n        enable_dhcp = true\n    }\n\n    subnets = [\n        {\n            name = "web"\n            cidr = "10.0.1.0/24"\n            zone = "us-west-2a"\n            public = true\n        }\n        {\n            name = "app"\n            cidr = "10.0.2.0/24"\n            zone = "us-west-2b"\n            public = false\n        }\n        {\n            name = "data"\n            cidr = "10.0.3.0/24"\n            zone = "us-west-2c"\n            public = false\n        }\n    ]\n\n    security_groups = [\n        {\n            name = "web"\n            rules = [\n                {protocol = "tcp", port = 80, source = "0.0.0.0/0"}\n                {protocol = "tcp", port = 443, source = "0.0.0.0/0"}\n            ]\n        }\n        {\n            name = "app"\n            rules = [\n                {protocol = "tcp", port = 8080, source = "10.0.1.0/24"}\n            ]\n        }\n    ]\n\n    load_balancers = [\n        {\n            name = "web-lb"\n            type = "application"\n            scheme = "internet-facing"\n            subnets = ["web"]\n            targets = ["web-01", "web-02"]\n        }\n    ]\n}\n```\n\n### Network Commands\n\n```\n# Show network configuration\nprovisioning network show --infra my-infra\n\n# Create network resources\nprovisioning network create --infra my-infra\n\n# Update network configuration\nprovisioning network update --infra my-infra\n\n# Test network connectivity\nprovisioning network test --infra my-infra\n```\n\n## Storage Management\n\n### Storage Configuration\n\n```\n# Storage configuration\nstorage: {\n    # Block storage\n    volumes = [\n        {\n            name = "app-data"\n            size = "100 GB"\n            type = "gp3"\n            encrypted = true\n        }\n    ]\n\n    # Object storage\n    buckets = [\n        {\n            name = "app-assets"\n            region = "us-west-2"\n            versioning = true\n            encryption = "AES256"\n        }\n    ]\n\n    # Backup configuration\n    backup = {\n        schedule = "0 1 * * *"  # Daily at 1 AM\n        retention = {\n            daily = 7\n            weekly = 4\n            monthly = 12\n        }\n    }\n}\n```\n\n### Storage Commands\n\n```\n# Create storage resources\nprovisioning storage create --infra my-infra\n\n# List storage\nprovisioning storage list --infra my-infra\n\n# Backup data\nprovisioning storage backup --infra my-infra\n\n# Restore from backup\nprovisioning storage restore --backup latest --infra my-infra\n```\n\n## Monitoring and Observability\n\n### Monitoring Setup\n\n```\n# Install monitoring stack\nprovisioning taskserv create prometheus --infra my-infra\nprovisioning taskserv create grafana --infra my-infra\nprovisioning taskserv create alertmanager --infra my-infra\n\n# Configure monitoring\nprovisioning taskserv configure prometheus --config monitoring.yaml --infra my-infra\n```\n\n### Health Checks\n\n```\n# Check overall infrastructure health\nprovisioning health check --infra my-infra\n\n# Check specific components\nprovisioning health check servers --infra my-infra\nprovisioning health check taskservs --infra my-infra\nprovisioning health check clusters --infra my-infra\n\n# Continuous monitoring\nprovisioning health monitor --infra my-infra --watch\n```\n\n### Metrics and Alerting\n\n```\n# Get infrastructure metrics\nprovisioning metrics get --infra my-infra\n\n# Set up alerts\nprovisioning alerts create --config alerts.yaml --infra my-infra\n\n# List active alerts\nprovisioning alerts list --infra my-infra\n```\n\n## Cost Management\n\n### Cost Monitoring\n\n```\n# Show current costs\nprovisioning cost show --infra my-infra\n\n# Cost breakdown by component\nprovisioning cost breakdown --infra my-infra\n\n# Cost trends\nprovisioning cost trends --period 30d --infra my-infra\n\n# Set cost alerts\nprovisioning cost alert --threshold 1000 --infra my-infra\n```\n\n### Cost Optimization\n\n```\n# Analyze cost optimization opportunities\nprovisioning cost optimize --infra my-infra\n\n# Show unused resources\nprovisioning cost unused --infra my-infra\n\n# Right-size recommendations\nprovisioning cost recommendations --infra my-infra\n```\n\n## Scaling Strategies\n\n### Manual Scaling\n\n```\n# Scale servers\nprovisioning server scale --count 5 --infra my-infra\n\n# Scale specific service\nprovisioning taskserv scale kubernetes --nodes 3 --infra my-infra\n\n# Scale cluster\nprovisioning cluster scale web-cluster --replicas 10 --infra my-infra\n```\n\n### Auto-scaling Configuration\n\n```\n# Auto-scaling configuration\nauto_scaling: {\n    servers = {\n        min_count = 2\n        max_count = 10\n\n        # Scaling metrics\n        cpu_threshold = 70\n        memory_threshold = 80\n\n        # Scaling behavior\n        scale_up_cooldown = "5m"\n        scale_down_cooldown = "10m"\n    }\n\n    clusters = {\n        web_cluster = {\n            min_replicas = 3\n            max_replicas = 20\n            metrics = [\n                {type = "cpu", target = 70}\n                {type = "memory", target = 80}\n                {type = "requests", target = 1000}\n            ]\n        }\n    }\n}\n```\n\n## Disaster Recovery\n\n### Backup Strategies\n\n```\n# Full infrastructure backup\nprovisioning backup create --type full --infra my-infra\n\n# Incremental backup\nprovisioning backup create --type incremental --infra my-infra\n\n# Schedule automated backups\nprovisioning backup schedule --daily --time "02:00" --infra my-infra\n```\n\n### Recovery Procedures\n\n```\n# List available backups\nprovisioning backup list --infra my-infra\n\n# Restore infrastructure\nprovisioning restore --backup latest --infra my-infra\n\n# Partial restore\nprovisioning restore --backup latest --components servers --infra my-infra\n\n# Test restore (dry run)\nprovisioning restore --backup latest --test --infra my-infra\n```\n\n## Advanced Infrastructure Patterns\n\n### Multi-Region Deployment\n\n```\n# Multi-region configuration\nregions: {\n    primary = {\n        name = "us-west-2"\n        servers = ["web-01", "web-02", "db-01"]\n        availability_zones = ["us-west-2a", "us-west-2b"]\n    }\n\n    secondary = {\n        name = "us-east-1"\n        servers = ["web-03", "web-04", "db-02"]\n        availability_zones = ["us-east-1a", "us-east-1b"]\n    }\n\n    # Cross-region replication\n    replication = {\n        database = {\n            primary = "us-west-2"\n            replicas = ["us-east-1"]\n            sync_mode = "async"\n        }\n\n        storage = {\n            sync_schedule = "*/15 * * * *"  # Every 15 minutes\n        }\n    }\n}\n```\n\n### Blue-Green Deployment\n\n```\n# Create green environment\nprovisioning generate infra --from production --name production-green\n\n# Deploy to green\nprovisioning server create --infra production-green\nprovisioning taskserv create --infra production-green\nprovisioning cluster deploy --infra production-green\n\n# Switch traffic to green\nprovisioning network switch --from production --to production-green\n\n# Decommission blue\nprovisioning server delete --infra production --yes\n```\n\n### Canary Deployment\n\n```\n# Create canary environment\nprovisioning cluster create web-cluster-canary --replicas 1 --infra my-infra\n\n# Route small percentage of traffic\nprovisioning network route --target web-cluster-canary --weight 10 --infra my-infra\n\n# Monitor canary metrics\nprovisioning metrics monitor web-cluster-canary --infra my-infra\n\n# Promote or rollback\nprovisioning cluster promote web-cluster-canary --infra my-infra\n# or\nprovisioning cluster rollback web-cluster-canary --infra my-infra\n```\n\n## Troubleshooting Infrastructure\n\n### Common Issues\n\n#### Server Creation Failures\n\n```\n# Check provider status\nprovisioning provider status aws\n\n# Validate server configuration\nprovisioning server validate web-01 --infra my-infra\n\n# Check quota limits\nprovisioning provider quota --infra my-infra\n\n# Debug server creation\nprovisioning --debug server create web-01 --infra my-infra\n```\n\n#### Service Installation Failures\n\n```\n# Check service prerequisites\nprovisioning taskserv check kubernetes --infra my-infra\n\n# Validate service configuration\nprovisioning taskserv validate kubernetes --infra my-infra\n\n# Check service logs\nprovisioning taskserv logs kubernetes --infra my-infra\n\n# Debug service installation\nprovisioning --debug taskserv create kubernetes --infra my-infra\n```\n\n#### Network Connectivity Issues\n\n```\n# Test network connectivity\nprovisioning network test --infra my-infra\n\n# Check security groups\nprovisioning network security-groups --infra my-infra\n\n# Trace network path\nprovisioning network trace --from web-01 --to db-01 --infra my-infra\n```\n\n### Performance Optimization\n\n```\n# Analyze performance bottlenecks\nprovisioning performance analyze --infra my-infra\n\n# Get performance recommendations\nprovisioning performance recommendations --infra my-infra\n\n# Monitor resource utilization\nprovisioning performance monitor --infra my-infra --duration 1h\n```\n\n## Testing Infrastructure\n\nThe provisioning system includes a comprehensive **Test Environment Service** for automated testing of infrastructure components before deployment.\n\n### Why Test Infrastructure\n\nTesting infrastructure before production deployment helps:\n\n- **Validate taskserv configurations** before installing on production servers\n- **Test integration** between multiple taskservs\n- **Verify cluster topologies** (Kubernetes, etcd, etc.) before deployment\n- **Catch configuration errors** early in the development cycle\n- **Ensure compatibility** between components\n\n### Test Environment Types\n\n#### 1. Single Taskserv Testing\n\nTest individual taskservs in isolated containers:\n\n```\n# Quick test (create, run, cleanup automatically)\nprovisioning test quick kubernetes\n\n# Single taskserv with custom resources\nprovisioning test env single postgres \\n  --cpu 2000 \\n  --memory 4096 \\n  --auto-start \\n  --auto-cleanup\n\n# Test with specific infrastructure context\nprovisioning test env single redis --infra my-infra\n```\n\n#### 2. Server Simulation\n\nTest complete server configurations with multiple taskservs:\n\n```\n# Simulate web server with multiple taskservs\nprovisioning test env server web-01 [containerd kubernetes cilium] \\n  --auto-start\n\n# Simulate database server\nprovisioning test env server db-01 [postgres redis] \\n  --infra prod-stack \\n  --auto-start\n```\n\n#### 3. Multi-Node Cluster Testing\n\nTest complex cluster topologies before production deployment:\n\n```\n# Test 3-node Kubernetes cluster\nprovisioning test topology load kubernetes_3node | \\n  test env cluster kubernetes --auto-start\n\n# Test etcd cluster\nprovisioning test topology load etcd_cluster | \\n  test env cluster etcd --auto-start\n\n# Test single-node Kubernetes\nprovisioning test topology load kubernetes_single | \\n  test env cluster kubernetes --auto-start\n```\n\n### Managing Test Environments\n\n```\n# List all test environments\nprovisioning test env list\n\n# Check environment status\nprovisioning test env status <env-id>\n\n# View environment logs\nprovisioning test env logs <env-id>\n\n# Cleanup environment when done\nprovisioning test env cleanup <env-id>\n```\n\n### Available Topology Templates\n\nPre-configured multi-node cluster templates:\n\n| Template | Description | Use Case |\n| ---------- | ------------- | ---------- |\n| `kubernetes_3node` | 3-node HA K8s cluster | Production-like K8s testing |\n| `kubernetes_single` | All-in-one K8s node | Development K8s testing |\n| `etcd_cluster` | 3-member etcd cluster | Distributed consensus testing |\n| `containerd_test` | Standalone containerd | Container runtime testing |\n| `postgres_redis` | Database stack | Database integration testing |\n\n### Test Environment Workflow\n\nTypical testing workflow:\n\n```\n# 1. Test new taskserv before deploying\nprovisioning test quick kubernetes\n\n# 2. If successful, test server configuration\nprovisioning test env server k8s-node [containerd kubernetes cilium] \\n  --auto-start\n\n# 3. Test complete cluster topology\nprovisioning test topology load kubernetes_3node | \\n  test env cluster kubernetes --auto-start\n\n# 4. Deploy to production\nprovisioning server create --infra production\nprovisioning taskserv create kubernetes --infra production\n```\n\n### CI/CD Integration\n\nIntegrate infrastructure testing into CI/CD pipelines:\n\n```\n# GitLab CI example\ntest-infrastructure:\n  stage: test\n  script:\n    # Start orchestrator\n    - ./scripts/start-orchestrator.nu --background\n\n    # Test critical taskservs\n    - provisioning test quick kubernetes\n    - provisioning test quick postgres\n    - provisioning test quick redis\n\n    # Test cluster topology\n    - provisioning test topology load kubernetes_3node |\n        test env cluster kubernetes --auto-start\n\n  artifacts:\n    when: on_failure\n    paths:\n      - test-logs/\n```\n\n### Prerequisites\n\nTest environments require:\n\n1. **Docker Running**: Test environments use Docker containers\n\n   ```bash\n   docker ps  # Should work without errors\n   ```\n\n1. **Orchestrator Running**: The orchestrator manages test containers\n\n   ```bash\n   cd provisioning/platform/orchestrator\n   ./scripts/start-orchestrator.nu --background\n   ```\n\n### Advanced Testing\n\n#### Custom Topology Testing\n\nCreate custom topology configurations:\n\n```\n# custom-topology.toml\n[my_cluster]\nname = "Custom Test Cluster"\ncluster_type = "custom"\n\n[[my_cluster.nodes]]\nname = "node-01"\nrole = "primary"\ntaskservs = ["postgres", "redis"]\n[my_cluster.nodes.resources]\ncpu_millicores = 2000\nmemory_mb = 4096\n\n[[my_cluster.nodes]]\nname = "node-02"\nrole = "replica"\ntaskservs = ["postgres"]\n[my_cluster.nodes.resources]\ncpu_millicores = 1000\nmemory_mb = 2048\n```\n\nLoad and test custom topology:\n\n```\nprovisioning test env cluster custom-app custom-topology.toml --auto-start\n```\n\n#### Integration Testing\n\nTest taskserv dependencies:\n\n```\n# Test Kubernetes dependencies in order\nprovisioning test quick containerd\nprovisioning test quick etcd\nprovisioning test quick kubernetes\nprovisioning test quick cilium\n\n# Test complete stack\nprovisioning test env server k8s-stack \\n  [containerd etcd kubernetes cilium] \\n  --auto-start\n```\n\n### Documentation\n\nFor complete test environment documentation:\n\n- **Test Environment Guide**: `docs/user/test-environment-guide.md`\n- **Detailed Usage**: `docs/user/test-environment-usage.md`\n- **Orchestrator README**: `provisioning/platform/orchestrator/README.md`\n\n## Best Practices\n\n### 1. Infrastructure Design\n\n- **Principle of Least Privilege**: Grant minimal necessary access\n- **Defense in Depth**: Multiple layers of security\n- **High Availability**: Design for failure resilience\n- **Scalability**: Plan for growth from the start\n\n### 2. Operational Excellence\n\n```\n# Always validate before applying changes\nprovisioning validate config --infra my-infra\n\n# Use check mode for dry runs\nprovisioning server create --check --infra my-infra\n\n# Monitor continuously\nprovisioning health monitor --infra my-infra\n\n# Regular backups\nprovisioning backup schedule --daily --infra my-infra\n```\n\n### 3. Security\n\n```\n# Regular security updates\nprovisioning taskserv update --security-only --infra my-infra\n\n# Encrypt sensitive data\nprovisioning sops settings.ncl --infra my-infra\n\n# Audit access\nprovisioning audit logs --infra my-infra\n```\n\n### 4. Cost Optimization\n\n```\n# Regular cost reviews\nprovisioning cost analyze --infra my-infra\n\n# Right-size resources\nprovisioning cost optimize --apply --infra my-infra\n\n# Use reserved instances for predictable workloads\nprovisioning server reserve --infra my-infra\n```\n\n## Next Steps\n\nNow that you understand infrastructure management:\n\n1. **Learn about extensions**: [Extension Development Guide](extension-development.md)\n2. **Master configuration**: [Configuration Guide](configuration.md)\n3. **Explore advanced examples**: [Examples and Tutorials](examples/)\n4. **Set up monitoring and alerting**\n5. **Implement automated scaling**\n6. **Plan disaster recovery procedures**\n\nYou now have the knowledge to build and manage robust, scalable cloud infrastructure!
