# Update Existing Infrastructure\n\n**Goal**: Safely update running infrastructure with minimal downtime\n**Time**: 15-30 minutes\n**Difficulty**: Intermediate\n\n## Overview\n\nThis guide covers:\n\n1. Checking for updates\n2. Planning update strategies\n3. Updating task services\n4. Rolling updates\n5. Rollback procedures\n6. Verification\n\n## Update Strategies\n\n### Strategy 1: In-Place Updates (Fastest)\n\n**Best for**: Non-critical environments, development, staging\n\n```\n# Direct update without downtime consideration\nprovisioning t create --infra \n```\n\n### Strategy 2: Rolling Updates (Recommended)\n\n**Best for**: Production environments, high availability\n\n```\n# Update servers one by one\nprovisioning s update --infra --rolling\n```\n\n### Strategy 3: Blue-Green Deployment (Safest)\n\n**Best for**: Critical production, zero-downtime requirements\n\n```\n# Create new infrastructure, switch traffic, remove old\nprovisioning ws init -green\n# ... configure and deploy\n# ... switch traffic\nprovisioning ws delete -blue\n```\n\n## Step 1: Check for Updates\n\n### 1.1 Check All Task Services\n\n```\n# Check all taskservs for updates\nprovisioning t check-updates\n```\n\n**Expected Output:**\n\n```\n📦 Task Service Update Check:\n\nNAME CURRENT LATEST STATUS\nkubernetes 1.29.0 1.30.0 ⬆️ update available\ncontainerd 1.7.13 1.7.13 ✅ up-to-date\ncilium 1.14.5 1.15.0 ⬆️ update available\npostgres 15.5 16.1 ⬆️ update available\nredis 7.2.3 7.2.3 ✅ up-to-date\n\nUpdates available: 3\n```\n\n### 1.2 Check Specific Task Service\n\n```\n# Check specific taskserv\nprovisioning t check-updates kubernetes\n```\n\n**Expected Output:**\n\n```\n📦 Kubernetes Update Check:\n\nCurrent: 1.29.0\nLatest: 1.30.0\nStatus: ⬆️ Update available\n\nChangelog:\n • Enhanced security features\n • Performance improvements\n • Bug fixes in kube-apiserver\n • New workload resource types\n\nBreaking Changes:\n • None\n\nRecommended: ✅ Safe to update\n```\n\n### 1.3 Check Version Status\n\n```\n# Show detailed version information\nprovisioning version show\n```\n\n**Expected Output:**\n\n```\n📋 Component Versions:\n\nCOMPONENT CURRENT LATEST DAYS OLD STATUS\nkubernetes 1.29.0 1.30.0 45 ⬆️ update\ncontainerd 1.7.13 1.7.13 0 ✅ current\ncilium 1.14.5 1.15.0 30 ⬆️ update\npostgres 15.5 16.1 60 ⬆️ update (major)\nredis 7.2.3 7.2.3 0 ✅ current\n```\n\n### 1.4 Check for Security Updates\n\n```\n# Check for security-related updates\nprovisioning version updates --security-only\n```\n\n## Step 2: Plan Your Update\n\n### 2.1 Review Current Configuration\n\n```\n# Show current infrastructure\nprovisioning show settings --infra my-production\n```\n\n### 2.2 Backup Configuration\n\n```\n# Create configuration backup\ncp -r workspace/infra/my-production workspace/infra/my-production.backup-$(date +%Y%m%d)\n\n# Or use built-in backup\nprovisioning ws backup my-production\n```\n\n**Expected Output:**\n\n```\n✅ Backup created: workspace/backups/my-production-20250930.tar.gz\n```\n\n### 2.3 Create Update Plan\n\n```\n# Generate update plan\nprovisioning plan update --infra my-production\n```\n\n**Expected Output:**\n\n```\n📝 Update Plan for my-production:\n\nPhase 1: Minor Updates (Low Risk)\n • containerd: No update needed\n • redis: No update needed\n\nPhase 2: Patch Updates (Medium Risk)\n • cilium: 1.14.5 → 1.15.0 (estimated 5 minutes)\n\nPhase 3: Major Updates (High Risk - Requires Testing)\n • kubernetes: 1.29.0 → 1.30.0 (estimated 15 minutes)\n • postgres: 15.5 → 16.1 (estimated 10 minutes, may require data migration)\n\nRecommended Order:\n 1. Update cilium (low risk)\n 2. Update kubernetes (test in staging first)\n 3. Update postgres (requires maintenance window)\n\nTotal Estimated Time: 30 minutes\nRecommended: Test in staging environment first\n```\n\n## Step 3: Update Task Services\n\n### 3.1 Update Non-Critical Service (Cilium Example)\n\n#### Dry-Run Update\n\n```\n# Test update without applying\nprovisioning t create cilium --infra my-production --check\n```\n\n**Expected Output:**\n\n```\n🔍 CHECK MODE: Simulating Cilium update\n\nCurrent: 1.14.5\nTarget: 1.15.0\n\nWould perform:\n 1. Download Cilium 1.15.0\n 2. Update configuration\n 3. Rolling restart of Cilium pods\n 4. Verify connectivity\n\nEstimated downtime: <1 minute per node\nNo errors detected. Ready to update.\n```\n\n#### Generate Updated Configuration\n\n```\n# Generate new configuration\nprovisioning t generate cilium --infra my-production\n```\n\n**Expected Output:**\n\n```\n✅ Generated Cilium configuration (version 1.15.0)\n Saved to: workspace/infra/my-production/taskservs/cilium.ncl\n```\n\n#### Apply Update\n\n```\n# Apply update\nprovisioning t create cilium --infra my-production\n```\n\n**Expected Output:**\n\n```\n🚀 Updating Cilium on my-production...\n\nDownloading Cilium 1.15.0... ⏳\n✅ Downloaded\n\nUpdating configuration... ⏳\n✅ Configuration updated\n\nRolling restart: web-01... ⏳\n✅ web-01 updated (Cilium 1.15.0)\n\nRolling restart: web-02... ⏳\n✅ web-02 updated (Cilium 1.15.0)\n\nVerifying connectivity... ⏳\n✅ All nodes connected\n\n🎉 Cilium update complete!\n Version: 1.14.5 → 1.15.0\n Downtime: 0 minutes\n```\n\n#### Verify Update\n\n```\n# Verify updated version\nprovisioning version taskserv cilium\n```\n\n**Expected Output:**\n\n```\n📦 Cilium Version Info:\n\nInstalled: 1.15.0\nLatest: 1.15.0\nStatus: ✅ Up-to-date\n\nNodes:\n ✅ web-01: 1.15.0 (running)\n ✅ web-02: 1.15.0 (running)\n```\n\n### 3.2 Update Critical Service (Kubernetes Example)\n\n#### Test in Staging First\n\n```\n# If you have staging environment\nprovisioning t create kubernetes --infra my-staging --check\nprovisioning t create kubernetes --infra my-staging\n\n# Run integration tests\nprovisioning test kubernetes --infra my-staging\n```\n\n#### Backup Current State\n\n```\n# Backup Kubernetes state\nkubectl get all -A -o yaml > k8s-backup-$(date +%Y%m%d).yaml\n\n# Backup etcd (if using external etcd)\nprovisioning t backup kubernetes --infra my-production\n```\n\n#### Schedule Maintenance Window\n\n```\n# Set maintenance mode (optional, if supported)\nprovisioning maintenance enable --infra my-production --duration 30m\n```\n\n#### Update Kubernetes\n\n```\n# Update control plane first\nprovisioning t create kubernetes --infra my-production --control-plane-only\n```\n\n**Expected Output:**\n\n```\n🚀 Updating Kubernetes control plane on my-production...\n\nDraining control plane: web-01... ⏳\n✅ web-01 drained\n\nUpdating control plane: web-01... ⏳\n✅ web-01 updated (Kubernetes 1.30.0)\n\nUncordoning: web-01... ⏳\n✅ web-01 ready\n\nVerifying control plane... ⏳\n✅ Control plane healthy\n\n🎉 Control plane update complete!\n```\n\n```\n# Update worker nodes one by one\nprovisioning t create kubernetes --infra my-production --workers-only --rolling\n```\n\n**Expected Output:**\n\n```\n🚀 Updating Kubernetes workers on my-production...\n\nRolling update: web-02...\n Draining... ⏳\n ✅ Drained (pods rescheduled)\n\n Updating... ⏳\n ✅ Updated (Kubernetes 1.30.0)\n\n Uncordoning... ⏳\n ✅ Ready\n\n Waiting for pods to stabilize... ⏳\n ✅ All pods running\n\n🎉 Worker update complete!\n Updated: web-02\n Version: 1.30.0\n```\n\n#### Verify Update\n\n```\n# Verify Kubernetes cluster\nkubectl get nodes\nprovisioning version taskserv kubernetes\n```\n\n**Expected Output:**\n\n```\nNAME STATUS ROLES AGE VERSION\nweb-01 Ready control-plane 30d v1.30.0\nweb-02 Ready 30d v1.30.0\n```\n\n```\n# Run smoke tests\nprovisioning test kubernetes --infra my-production\n```\n\n### 3.3 Update Database (PostgreSQL Example)\n\n⚠️ **WARNING**: Database updates may require data migration. Always backup first!\n\n#### Backup Database\n\n```\n# Backup PostgreSQL database\nprovisioning t backup postgres --infra my-production\n```\n\n**Expected Output:**\n\n```\n🗄️ Backing up PostgreSQL...\n\nCreating dump: my-production-postgres-20250930.sql... ⏳\n✅ Dump created (2.3 GB)\n\nCompressing... ⏳\n✅ Compressed (450 MB)\n\nSaved to: workspace/backups/postgres/my-production-20250930.sql.gz\n```\n\n#### Check Compatibility\n\n```\n# Check if data migration is needed\nprovisioning t check-migration postgres --from 15.5 --to 16.1\n```\n\n**Expected Output:**\n\n```\n🔍 PostgreSQL Migration Check:\n\nFrom: 15.5\nTo: 16.1\n\nMigration Required: ✅ Yes (major version change)\n\nSteps Required:\n 1. Dump database with pg_dump\n 2. Stop PostgreSQL 15.5\n 3. Install PostgreSQL 16.1\n 4. Initialize new data directory\n 5. Restore from dump\n\nEstimated Time: 15-30 minutes (depending on data size)\nEstimated Downtime: 15-30 minutes\n\nRecommended: Use streaming replication for zero-downtime upgrade\n```\n\n#### Perform Update\n\n```\n# Update PostgreSQL (with automatic migration)\nprovisioning t create postgres --infra my-production --migrate\n```\n\n**Expected Output:**\n\n```\n🚀 Updating PostgreSQL on my-production...\n\n⚠️ Major version upgrade detected (15.5 → 16.1)\n Automatic migration will be performed\n\nDumping database... ⏳\n✅ Database dumped (2.3 GB)\n\nStopping PostgreSQL 15.5... ⏳\n✅ Stopped\n\nInstalling PostgreSQL 16.1... ⏳\n✅ Installed\n\nInitializing new data directory... ⏳\n✅ Initialized\n\nRestoring database... ⏳\n✅ Restored (2.3 GB)\n\nStarting PostgreSQL 16.1... ⏳\n✅ Started\n\nVerifying data integrity... ⏳\n✅ All tables verified\n\n🎉 PostgreSQL update complete!\n Version: 15.5 → 16.1\n Downtime: 18 minutes\n```\n\n#### Verify Update\n\n```\n# Verify PostgreSQL\nprovisioning version taskserv postgres\nssh db-01 "psql --version"\n```\n\n## Step 4: Update Multiple Services\n\n### 4.1 Batch Update (Sequentially)\n\n```\n# Update multiple taskservs one by one\nprovisioning t update --infra my-production --taskservs cilium,containerd,redis\n```\n\n**Expected Output:**\n\n```\n🚀 Updating 3 taskservs on my-production...\n\n[1/3] Updating cilium... ⏳\n✅ cilium updated (1.15.0)\n\n[2/3] Updating containerd... ⏳\n✅ containerd updated (1.7.14)\n\n[3/3] Updating redis... ⏳\n✅ redis updated (7.2.4)\n\n🎉 All updates complete!\n Updated: 3 taskservs\n Total time: 8 minutes\n```\n\n### 4.2 Parallel Update (Non-Dependent Services)\n\n```\n# Update taskservs in parallel (if they don't depend on each other)\nprovisioning t update --infra my-production --taskservs redis,postgres --parallel\n```\n\n**Expected Output:**\n\n```\n🚀 Updating 2 taskservs in parallel on my-production...\n\nredis: Updating... ⏳\npostgres: Updating... ⏳\n\nredis: ✅ Updated (7.2.4)\npostgres: ✅ Updated (16.1)\n\n🎉 All updates complete!\n Updated: 2 taskservs\n Total time: 3 minutes (parallel)\n```\n\n## Step 5: Update Server Configuration\n\n### 5.1 Update Server Resources\n\n```\n# Edit server configuration\nprovisioning sops workspace/infra/my-production/servers.ncl\n```\n\n**Example: Upgrade server plan**\n\n```\n# Before\n{\n name = "web-01"\n plan = "1xCPU-2 GB" # Old plan\n}\n\n# After\n{\n name = "web-01"\n plan = "2xCPU-4 GB" # New plan\n}\n```\n\n```\n# Apply server update\nprovisioning s update --infra my-production --check\nprovisioning s update --infra my-production\n```\n\n### 5.2 Update Server OS\n\n```\n# Update operating system packages\nprovisioning s update --infra my-production --os-update\n```\n\n**Expected Output:**\n\n```\n🚀 Updating OS packages on my-production servers...\n\nweb-01: Updating packages... ⏳\n✅ web-01: 24 packages updated\n\nweb-02: Updating packages... ⏳\n✅ web-02: 24 packages updated\n\ndb-01: Updating packages... ⏳\n✅ db-01: 24 packages updated\n\n🎉 OS updates complete!\n```\n\n## Step 6: Rollback Procedures\n\n### 6.1 Rollback Task Service\n\nIf update fails or causes issues:\n\n```\n# Rollback to previous version\nprovisioning t rollback cilium --infra my-production\n```\n\n**Expected Output:**\n\n```\n🔄 Rolling back Cilium on my-production...\n\nCurrent: 1.15.0\nTarget: 1.14.5 (previous version)\n\nRolling back: web-01... ⏳\n✅ web-01 rolled back\n\nRolling back: web-02... ⏳\n✅ web-02 rolled back\n\nVerifying connectivity... ⏳\n✅ All nodes connected\n\n🎉 Rollback complete!\n Version: 1.15.0 → 1.14.5\n```\n\n### 6.2 Rollback from Backup\n\n```\n# Restore configuration from backup\nprovisioning ws restore my-production --from workspace/backups/my-production-20250930.tar.gz\n```\n\n### 6.3 Emergency Rollback\n\n```\n# Complete infrastructure rollback\nprovisioning rollback --infra my-production --to-snapshot \n```\n\n## Step 7: Post-Update Verification\n\n### 7.1 Verify All Components\n\n```\n# Check overall health\nprovisioning health --infra my-production\n```\n\n**Expected Output:**\n\n```\n🏥 Health Check: my-production\n\nServers:\n ✅ web-01: Healthy\n ✅ web-02: Healthy\n ✅ db-01: Healthy\n\nTask Services:\n ✅ kubernetes: 1.30.0 (healthy)\n ✅ containerd: 1.7.13 (healthy)\n ✅ cilium: 1.15.0 (healthy)\n ✅ postgres: 16.1 (healthy)\n\nClusters:\n ✅ buildkit: 2/2 replicas (healthy)\n\nOverall Status: ✅ All systems healthy\n```\n\n### 7.2 Verify Version Updates\n\n```\n# Verify all versions are updated\nprovisioning version show\n```\n\n### 7.3 Run Integration Tests\n\n```\n# Run comprehensive tests\nprovisioning test all --infra my-production\n```\n\n**Expected Output:**\n\n```\n🧪 Running Integration Tests...\n\n[1/5] Server connectivity... ⏳\n✅ All servers reachable\n\n[2/5] Kubernetes health... ⏳\n✅ All nodes ready, all pods running\n\n[3/5] Network connectivity... ⏳\n✅ All services reachable\n\n[4/5] Database connectivity... ⏳\n✅ PostgreSQL responsive\n\n[5/5] Application health... ⏳\n✅ All applications healthy\n\n🎉 All tests passed!\n```\n\n### 7.4 Monitor for Issues\n\n```\n# Monitor logs for errors\nprovisioning logs --infra my-production --follow --level error\n```\n\n## Update Checklist\n\nUse this checklist for production updates:\n\n- [ ] Check for available updates\n- [ ] Review changelog and breaking changes\n- [ ] Create configuration backup\n- [ ] Test update in staging environment\n- [ ] Schedule maintenance window\n- [ ] Notify team/users of maintenance\n- [ ] Update non-critical services first\n- [ ] Verify each update before proceeding\n- [ ] Update critical services with rolling updates\n- [ ] Backup database before major updates\n- [ ] Verify all components after update\n- [ ] Run integration tests\n- [ ] Monitor for issues (30 minutes minimum)\n- [ ] Document any issues encountered\n- [ ] Close maintenance window\n\n## Common Update Scenarios\n\n### Scenario 1: Minor Security Patch\n\n```\n# Quick security update\nprovisioning t check-updates --security-only\nprovisioning t update --infra my-production --security-patches --yes\n```\n\n### Scenario 2: Major Version Upgrade\n\n```\n# Careful major version update\nprovisioning ws backup my-production\nprovisioning t check-migration --from X.Y --to X+1.Y\nprovisioning t create --infra my-production --migrate\nprovisioning test all --infra my-production\n```\n\n### Scenario 3: Emergency Hotfix\n\n```\n# Apply critical hotfix immediately\nprovisioning t create --infra my-production --hotfix --yes\n```\n\n## Troubleshooting Updates\n\n### Issue: Update fails mid-process\n\n**Solution:**\n\n```\n# Check update status\nprovisioning t status --infra my-production\n\n# Resume failed update\nprovisioning t update --infra my-production --resume\n\n# Or rollback\nprovisioning t rollback --infra my-production\n```\n\n### Issue: Service not starting after update\n\n**Solution:**\n\n```\n# Check logs\nprovisioning logs --infra my-production\n\n# Verify configuration\nprovisioning t validate --infra my-production\n\n# Rollback if necessary\nprovisioning t rollback --infra my-production\n```\n\n### Issue: Data migration fails\n\n**Solution:**\n\n```\n# Check migration logs\nprovisioning t migration-logs --infra my-production\n\n# Restore from backup\nprovisioning t restore --infra my-production --from \n```\n\n## Best Practices\n\n1. **Always Test First**: Test updates in staging before production\n2. **Backup Everything**: Create backups before any update\n3. **Update Gradually**: Update one service at a time\n4. **Monitor Closely**: Watch for errors after each update\n5. **Have Rollback Plan**: Always have a rollback strategy\n6. **Document Changes**: Keep update logs for reference\n7. **Schedule Wisely**: Update during low-traffic periods\n8. **Verify Thoroughly**: Run tests after each update\n\n## Next Steps\n\n- **[Customize Guide](customize-infrastructure.md)** - Customize your infrastructure\n- **[From Scratch Guide](from-scratch.md)** - Deploy new infrastructure\n- **[Workflow Guide](../development/workflow.md)** - Automate with workflows\n\n## Quick Reference\n\n```\n# Update workflow\nprovisioning t check-updates\nprovisioning ws backup my-production\nprovisioning t create --infra my-production --check\nprovisioning t create --infra my-production\nprovisioning version taskserv \nprovisioning health --infra my-production\nprovisioning test all --infra my-production\n```\n\n---\n\n*This guide is part of the provisioning project documentation. Last updated: 2025-09-30*