# Upgrading Provisioning

Upgrade Provisioning to a new version with minimal downtime and automatic rollback support.

## Overview

Provisioning supports two upgrade strategies:

1. **In-Place Upgrade** - Update existing installation
2. **Side-by-Side Upgrade** - Run new version alongside old, switch when ready

Both strategies support automatic rollback on failure.

## Before Upgrading

### Check Current Version

```bash
provisioning version

# Example output:
# Provisioning v5.0.0
# Nushell 0.109.0
# Nickel 1.15.1
# SOPS 3.10.2
# Age 1.2.1
```

### Backup Configuration

```bash
# Backup entire workspace
provisioning workspace backup

# Backup specific configuration
provisioning config backup

# Backup state
provisioning state backup
```

### Check Changelog

```bash
# View latest changes
provisioning changelog

# Check upgrade path
provisioning version --check-upgrade

# Show upgrade recommendations
provisioning upgrade --check
```

### Verify System Health

```bash
# Health check
provisioning health check

# Check all services
provisioning platform health

# Verify provider connectivity
provisioning providers test --all

# Validate configuration
provisioning validate config --strict
```

## Upgrade Methods

### Method 1: In-Place Upgrade

Upgrade the existing installation with zero downtime:

```bash
# Check upgrade compatibility
provisioning upgrade --check

# List breaking changes
provisioning upgrade --breaking-changes

# Show migration guide (if any)
provisioning upgrade --show-migration

# Perform upgrade
provisioning upgrade
```

**Process**:
1. Validate current installation
2. Download new version
3. Run migration scripts (if needed)
4. Restart services
5. Verify health
6. Keep old version for rollback (24 hours)

### Method 2: Side-by-Side Upgrade

Run new version alongside old version for testing:

```bash
# Create staging installation
provisioning upgrade --staging --version v5.1.0

# Test new version
provisioning --staging server list

# Run test suite
provisioning --staging test suite

# Switch to new version
provisioning upgrade --activate

# Remove old version (after confirmation)
provisioning upgrade --cleanup-old
```

**Advantages**:
- Test new version before switching
- Zero downtime during upgrade
- Easy rollback to previous version
- Run both versions simultaneously

## Upgrade Process

### Step 1: Pre-Upgrade Checks

```bash
# Check system requirements
provisioning setup validate

# Verify dependencies are up-to-date
provisioning version --check-dependencies

# Check disk space (minimum 2GB required)
df -h /

# Verify all services healthy
provisioning platform health
```

### Step 2: Backup Data

```bash
# Backup entire workspace
provisioning workspace backup --compress

# Backup orchestrator state
provisioning orchestrator backup

# Backup configuration
provisioning config backup

# Verify backup
provisioning backup list
provisioning backup verify --latest
```

### Step 3: Download New Version

```bash
# Check available versions
provisioning version --available

# Download specific version
provisioning upgrade --download v5.1.0

# Verify download
provisioning upgrade --verify-download v5.1.0

# Check size
provisioning upgrade --show-size v5.1.0
```

### Step 4: Run Migration Scripts

```bash
# Show required migrations
provisioning upgrade --show-migrations

# Test migration (dry-run)
provisioning upgrade --dry-run

# Run migrations
provisioning upgrade --migrate

# Verify migration
provisioning upgrade --verify-migration
```

### Step 5: Perform Upgrade

```bash
# Stop orchestrator gracefully
provisioning orchestrator stop --graceful

# Install new version
provisioning upgrade --install

# Verify installation
provisioning version
provisioning validate config

# Start services
provisioning orchestrator start
```

### Step 6: Verify Upgrade

```bash
# Check version
provisioning version

# Health check
provisioning health check

# Run test suite
provisioning test quick

# Verify provider connectivity
provisioning providers test --all

# Check orchestrator status
provisioning orchestrator status
```

## Breaking Changes

Some upgrades may include breaking changes. Check before upgrading:

```bash
# List breaking changes
provisioning upgrade --breaking-changes

# Show migration guide
provisioning upgrade --migration-guide v5.1.0

# Generate migration script
provisioning upgrade --generate-migration v5.1.0 > migrate.nu
```

## Common Migration Scenarios

### Scenario 1: Configuration Format Change

If configuration format changes (e.g., TOML → YAML):

```bash
# Export old format
provisioning config export --format toml > config.old.toml

# Run migration
provisioning upgrade --migrate-config

# Verify new format
provisioning config export --format yaml | head -20
```

### Scenario 2: Schema Updates

If infrastructure schemas change:

```bash
# Validate against new schema
nickel typecheck workspace/infra/*.ncl

# Update schemas if needed
provisioning upgrade --update-schemas

# Regenerate configurations
provisioning config regenerate

# Validate updated config
provisioning validate config --strict
```

### Scenario 3: Provider API Changes

If provider APIs change:

```bash
# Test provider connectivity with new version
provisioning providers test upcloud --verbose

# Check provider configuration
provisioning config show --section providers.upcloud

# Update provider configuration if needed
provisioning providers configure upcloud

# Verify connectivity
provisioning server list
```

## Rollback Procedure

### Automatic Rollback

If upgrade fails, automatic rollback occurs:

```bash
# Monitor rollback progress
provisioning upgrade --watch

# Check rollback status
provisioning upgrade --status

# View rollback logs
provisioning upgrade --logs
```

### Manual Rollback

If needed, manually rollback to previous version:

```bash
# List available versions for rollback
provisioning upgrade --rollback-candidates

# Rollback to specific version
provisioning upgrade --rollback v5.0.0

# Verify rollback
provisioning version
provisioning platform health

# Restore from backup
provisioning backup restore --backup-id=<id>
```

## Batch Workflow Handling

If you have running batch workflows:

```bash
# Check running workflows
provisioning workflow list --status running

# Graceful shutdown (wait for completion)
provisioning workflow shutdown --graceful

# Force shutdown (immediate)
provisioning workflow shutdown --force

# Resume workflows after upgrade
provisioning workflow resume
```

## Troubleshooting Upgrades

### Upgrade Hangs

```bash
# Check logs
tail -f ~/.provisioning/logs/upgrade.log

# Monitor process
provisioning upgrade --monitor

# Stop upgrade gracefully
provisioning upgrade --stop --graceful

# Force stop
provisioning upgrade --stop --force
```

### Migration Failure

```bash
# Check migration logs
provisioning upgrade --migration-logs

# Rollback to previous version
provisioning upgrade --rollback

# Restore from backup
provisioning backup restore
```

### Service Won't Start

```bash
# Check service logs
provisioning platform logs

# Verify configuration
provisioning validate config --strict

# Restore configuration from backup
provisioning config restore

# Restart services
provisioning orchestrator start
```

## Upgrade Scheduling

### Schedule Automated Upgrade

```bash
# Schedule upgrade for specific time
provisioning upgrade --schedule "2026-01-20T02:00:00"

# Schedule for next maintenance window
provisioning upgrade --schedule-next-maintenance

# Cancel scheduled upgrade
provisioning upgrade --cancel-scheduled
```

### Unattended Upgrade

For CI/CD environments:

```bash
# Non-interactive upgrade
provisioning upgrade --yes --no-confirm

# Upgrade with timeout
provisioning upgrade --timeout 3600

# Skip backup
provisioning upgrade --skip-backup

# Continue even if health checks fail
provisioning upgrade --force-upgrade
```

## Version Management

### Version Constraints

Pin versions for workspace reproducibility:

```nickel
# workspace/versions.ncl
{
  provisioning = "5.0.0"
  nushell = "0.109.0"
  nickel = "1.15.1"
  sops = "3.10.2"
  age = "1.2.1"
}
```

Enforce version constraints:

```bash
# Check version compliance
provisioning version --check-constraints

# Enforce constraint
provisioning version --strict-mode
```

### Vendor Versions

Pin provider and task service versions:

```nickel
# workspace/infra/versions.ncl
{
  providers = {
    upcloud = "2.0.0"
    aws = "5.0.0"
  }
  taskservs = {
    kubernetes = "1.28.0"
    postgres = "14.0"
  }
}
```

## Best Practices

### 1. Plan Upgrades

- Schedule during maintenance windows
- Test in staging first
- Communicate with team
- Have rollback plan ready

### 2. Backup Everything

```bash
# Complete backup before upgrade
provisioning workspace backup --compress
provisioning config backup
provisioning state backup
```

### 3. Test Before Upgrading

```bash
# Use side-by-side upgrade to test
provisioning upgrade --staging
provisioning test suite
```

### 4. Monitor After Upgrade

```bash
# Watch orchestrator
provisioning orchestrator status --watch

# Monitor platform health
provisioning platform monitor

# Check logs
tail -f ~/.provisioning/logs/provisioning.log
```

### 5. Document Changes

```bash
# Record what changed
provisioning upgrade --changelog > UPGRADE.md

# Update team documentation
# Update runbooks
# Update dashboards
```

## Upgrade Policies

### Automatic Updates

Enable automatic updates:

```yaml
# ~/.config/provisioning/user_config.yaml
upgrade:
  auto_update: true
  check_interval: "daily"
  update_channel: "stable"
  auto_backup: true
```

### Update Channels

Choose update channel:

```bash
# Stable releases (recommended)
provisioning upgrade --channel stable

# Beta releases
provisioning upgrade --channel beta

# Development (nightly)
provisioning upgrade --channel development
```

## Related Documentation

- [Initial Setup](../setup/initial-setup.md) - First-time configuration
- [Platform Health](platform-health.md) - System monitoring
- [Backup & Recovery](backup-recovery.md) - Data protection
