provisioning/docs/MIGRATION_EXAMPLE.md

# Migration Example: Complete Walkthrough

## Scenario

Migrating a production infrastructure project from the old `config.defaults.toml` system to the new workspace-based configuration.

**Current Setup**:
- Infrastructure name: `production-cluster`
- Providers: AWS, UpCloud
- Platform services: Orchestrator, Control Center, KMS
- Old config location: `/Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml`

**Target Setup**:
- Workspace name: `production-cluster`
- Workspace path: `~/workspaces/production-cluster`
- All configurations validated and tested

## Step-by-Step Migration

### Step 1: Pre-Migration Assessment

```bash
# Check current configuration
$ provisioning env

# Output:
PROVISIONING_BASE: /Users/Akasha/project-provisioning
PROVISIONING_CONFIG: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
PROVISIONING_PROVIDER: aws
...

# Review current config file
$ cat provisioning/config/config.defaults.toml | head -20

[core]
name = "provisioning"
version = "1.0.0"

[debug]
enabled = false
log_level = "info"

[providers]
default = "aws"
active = ["aws", "upcloud"]

[providers.aws]
region = "us-east-1"
...
```

### Step 2: Backup Current Configuration

```bash
# Create timestamped backup
$ cp -r provisioning/config provisioning/config.backup.$(date +%Y%m%d-%H%M%S)

# Verify backup
$ ls -la provisioning/config.backup.*

drwxr-xr-x  8 user  staff  256 Oct  6 10:30 provisioning/config.backup.20251006-103000
```

### Step 3: Dry Run Migration

```bash
# Preview migration without making changes
$ ./provisioning/scripts/migrate-to-target-configs.nu \
    --workspace-name "production-cluster" \
    --dry-run

🔄 Migration to Target-Based Configuration System
==================================================

⚠️  DRY RUN MODE - No changes will be made

Step 1: Detecting old configuration...
   Found: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
   ✅ Old config loaded

Step 3: Target workspace
   Name: production-cluster
   Path: /Users/Akasha/workspaces/production-cluster

   Would create workspace structure...

Step 5: Generating provisioning.yaml...
   Would write to: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
   Config preview:
workspace:
  name: production-cluster
  version: 1.0.0
  created: '2025-10-06T10:35:00Z'
paths:
  base: /Users/Akasha/workspaces/production-cluster
  infra: /Users/Akasha/workspaces/production-cluster/infra
  cache: /Users/Akasha/workspaces/production-cluster/.cache
  runtime: /Users/Akasha/workspaces/production-cluster/.runtime
core:
  name: provisioning
  version: 1.0.0
debug:
  enabled: false
  log_level: info
providers:
  active:
  - aws
  - upcloud
  default: aws
...

Step 6: Migrating provider configs...
   • Migrating aws...
     Would create: /Users/Akasha/workspaces/production-cluster/config/providers/aws.toml
   • Migrating upcloud...
     Would create: /Users/Akasha/workspaces/production-cluster/config/providers/upcloud.toml

Step 7: Creating user context...
   Would create: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml
```

### Step 4: Execute Migration with Backup

```bash
# Run actual migration
$ ./provisioning/scripts/migrate-to-target-configs.nu \
    --workspace-name "production-cluster" \
    --backup

🔄 Migration to Target-Based Configuration System
==================================================

Step 1: Detecting old configuration...
   Found: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
   ✅ Old config loaded

Step 2: Creating backup...
   ✅ Backup created: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml.backup.20251006-103500

Step 3: Target workspace
   Name: production-cluster
   Path: /Users/Akasha/workspaces/production-cluster

Step 4: Creating workspace...
   ✅ Created: /Users/Akasha/workspaces/production-cluster
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config/platform
   ✅ Created: /Users/Akasha/workspaces/production-cluster/infra
   ✅ Created: /Users/Akasha/workspaces/production-cluster/.cache
   ✅ Created: /Users/Akasha/workspaces/production-cluster/.runtime

Step 5: Generating provisioning.yaml...
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml

Step 6: Migrating provider configs...
   • Migrating aws...
     ✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers/aws.toml
   • Migrating upcloud...
     ✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers/upcloud.toml

Step 7: Creating user context...
   ✅ Created: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml

✅ Migration Complete!

📋 Summary:
   Workspace: production-cluster
   Path: /Users/Akasha/workspaces/production-cluster
   Config: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
   Context: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml

🎯 Next Steps:
   1. Review and customize: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
   2. Configure providers in: /Users/Akasha/workspaces/production-cluster/config/providers/
   3. Test: provisioning workspace config validate
   4. If all good, remove old config: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml

⚠️  IMPORTANT: Old config is still at /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
   Backup saved at: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml.backup.*
   Remove it manually after verifying migration
```

### Step 5: Verify Workspace Structure

```bash
# Check created workspace structure
$ tree ~/workspaces/production-cluster -L 3

/Users/Akasha/workspaces/production-cluster
├── config
│   ├── provisioning.yaml
│   ├── providers
│   │   ├── aws.toml
│   │   └── upcloud.toml
│   └── platform
├── infra
├── .cache
└── .runtime

# View main configuration
$ cat ~/workspaces/production-cluster/config/provisioning.yaml

workspace:
  name: production-cluster
  version: 1.0.0
  created: '2025-10-06T10:35:00Z'
paths:
  base: /Users/Akasha/workspaces/production-cluster
  infra: /Users/Akasha/workspaces/production-cluster/infra
  cache: /Users/Akasha/workspaces/production-cluster/.cache
  runtime: /Users/Akasha/workspaces/production-cluster/.runtime
...

# View provider configuration
$ cat ~/workspaces/production-cluster/config/providers/aws.toml

[provider]
name = "aws"
region = "us-east-1"
enabled = true

[credentials]
type = "environment"

[compute]
default_instance_type = "t3.medium"
...
```

### Step 6: Validate Configuration

```bash
# Validate workspace configuration
$ provisioning workspace config validate

✅ Validation passed

# Validate provider configurations
$ provisioning provider validate aws

✅ Validation passed

$ provisioning provider validate upcloud

✅ Validation passed
```

### Step 7: Run Validation Test Suite

```bash
# Run comprehensive validation tests
$ nu provisioning/tests/config_validation_tests.nu

🧪 Configuration Validation Test Suite
======================================

Test 1: Required Fields Validation
  ✅ PASSED

Test 2: Type Validation
  ✅ PASSED

Test 3: Enum Validation
  ✅ PASSED

Test 4: Range Validation
  ✅ PASSED

Test 5: Pattern Validation
  ✅ PASSED

Test 6: Deprecated Fields Warning
  ✅ PASSED

📊 Results: 6 passed, 0 failed
✅ All tests passed!
```

### Step 8: Test Operations

```bash
# Test server operations in check mode
$ provisioning --check server list

Checking server configuration...
✅ Configuration valid
✅ AWS provider configured
✅ UpCloud provider configured

# Test taskserv operations
$ provisioning --check taskserv list

Available taskservs:
  • kubernetes (v1.28.0)
  • containerd (v1.7.0)
  • etcd (v3.5.0)
  ...

# Verify workspace info
$ provisioning workspace info

Workspace: production-cluster
Path: /Users/Akasha/workspaces/production-cluster
Active: Yes
Version: 1.0.0
Created: 2025-10-06T10:35:00Z

Configuration:
  Providers: aws, upcloud
  Platform services: orchestrator, control-center, kms

Infrastructure:
  Count: 0
  Path: /Users/Akasha/workspaces/production-cluster/infra
```

### Step 9: Customize Configuration

```bash
# Edit main workspace config
$ vim ~/workspaces/production-cluster/config/provisioning.yaml

# Update debug settings for production
debug:
  enabled: false  # ← Ensure disabled in production
  log_level: "warn"  # ← Higher threshold

# Update output settings
output:
  format: "json"  # ← JSON for automation
  file_viewer: "jq"  # ← Better for JSON

# Save and validate
$ provisioning workspace config validate

✅ Validation passed

# Edit AWS provider config
$ vim ~/workspaces/production-cluster/config/providers/aws.toml

# Update region and credentials
[provider]
region = "eu-west-1"  # ← European region

[credentials]
type = "iam_role"  # ← Use IAM role in production

# Validate provider config
$ provisioning provider validate aws

✅ Validation passed
```

### Step 10: Create Platform Service Configs

```bash
# Create orchestrator config
$ cat > ~/workspaces/production-cluster/config/platform/orchestrator.toml << 'EOF'
[service]
name = "orchestrator"
enabled = true

[server]
host = "0.0.0.0"
port = 8080

[workers]
count = 4

[queue]
max_size = 1000
storage_path = "/Users/Akasha/workspaces/production-cluster/.runtime/queue"
EOF

# Validate orchestrator config
$ provisioning platform validate orchestrator

✅ Validation passed

# Create KMS config
$ cat > ~/workspaces/production-cluster/config/platform/kms.toml << 'EOF'
[kms]
enabled = true
provider = "aws_kms"

[encryption]
algorithm = "AES-256-GCM"
key_rotation_days = 90
EOF

# Validate KMS config
$ provisioning platform validate kms

✅ Validation passed
```

### Step 11: Update Infrastructure Definitions

```bash
# Copy existing infrastructure to new workspace
$ cp -r provisioning/infra/production-cluster \
       ~/workspaces/production-cluster/infra/

# Verify infrastructure
$ ls -la ~/workspaces/production-cluster/infra/production-cluster

total 32
-rw-r--r--  1 user  staff  1024 Oct  6 10:45 servers.yaml
-rw-r--r--  1 user  staff  2048 Oct  6 10:45 taskservs.yaml
-rw-r--r--  1 user  staff  1536 Oct  6 10:45 cluster.yaml
```

### Step 12: Final Validation

```bash
# Comprehensive validation
$ provisioning workspace config validate --verbose

Validating workspace configuration...
✅ Workspace name: production-cluster (valid)
✅ Version: 1.0.0 (valid semver)
✅ Paths configuration: valid
✅ Debug configuration: valid
✅ Output configuration: valid
✅ Provider configuration: valid
✅ Secrets configuration: valid

Validating providers...
✅ AWS provider: configured and valid
✅ UpCloud provider: configured and valid

Validating platform services...
✅ Orchestrator: configured and valid
✅ KMS: configured and valid

Overall: ✅ All validations passed
```

### Step 13: Test Deployment (Check Mode)

```bash
# Test server deployment in check mode
$ provisioning --check server create --infra production-cluster

Checking server deployment...
✅ Configuration loaded
✅ Provider validated (aws)
✅ Infrastructure definition found
✅ Resource limits checked
✅ Dependencies resolved

Would create:
  • web-01 (t3.medium, eu-west-1a)
  • web-02 (t3.medium, eu-west-1b)
  • db-01 (r5.large, eu-west-1a)

Dry run complete. Use without --check to deploy.
```

### Step 14: Clean Up Old Configuration

```bash
# Verify everything works with new configuration
$ provisioning workspace info
$ provisioning --check server list
$ provisioning --check taskserv list

# All good? Remove old config
$ rm provisioning/config/config.defaults.toml
$ rm provisioning/config/config.user.toml

# Keep backup for reference
$ ls -la provisioning/config.backup.*

-rw-r--r--  1 user  staff  8192 Oct  6 10:30 config.defaults.toml.backup.20251006-103500
```

### Step 15: Update CI/CD Pipeline

```yaml
# .gitlab-ci.yml
variables:
  PROVISIONING_WORKSPACE: "production-cluster"

stages:
  - validate
  - deploy

validate:
  stage: validate
  script:
    # Validate all configurations
    - provisioning workspace config validate
    - provisioning provider validate --all
    - provisioning platform validate --all
    # Run validation tests
    - nu provisioning/tests/config_validation_tests.nu
  only:
    changes:
      - "workspaces/*/config/**/*"

deploy:
  stage: deploy
  script:
    # Deploy with validated configuration
    - provisioning --check server create --infra production-cluster
    - provisioning server create --infra production-cluster
  only:
    - main
  when: manual
```

## Result

**Before Migration**:
- ❌ Monolithic config file
- ❌ No validation
- ❌ Hard to manage multiple environments
- ❌ Provider configs mixed with core settings

**After Migration**:
- ✅ Modular workspace structure
- ✅ Schema-based validation
- ✅ Clear separation of concerns
- ✅ Provider configs isolated
- ✅ Platform services configurable
- ✅ Test suite for validation
- ✅ CI/CD integration ready

## Workspace Final State

```
~/workspaces/production-cluster/
├── config/
│   ├── provisioning.yaml          # ✅ Main config (validated)
│   ├── providers/
│   │   ├── aws.toml               # ✅ AWS config (validated)
│   │   └── upcloud.toml           # ✅ UpCloud config (validated)
│   └── platform/
│       ├── orchestrator.toml      # ✅ Orchestrator (validated)
│       └── kms.toml               # ✅ KMS (validated)
├── infra/
│   └── production-cluster/        # ✅ Infrastructure definitions
│       ├── servers.yaml
│       ├── taskservs.yaml
│       └── cluster.yaml
├── .cache/                        # ✅ Cache directory
└── .runtime/                      # ✅ Runtime data

~/Library/Application Support/provisioning/
└── ws_production-cluster.yaml     # ✅ User context
```

## Commands Reference

### Migration
```bash
./provisioning/scripts/migrate-to-target-configs.nu --workspace-name "production-cluster" --backup
```

### Validation
```bash
provisioning workspace config validate
provisioning provider validate aws
provisioning platform validate orchestrator
```

### Testing
```bash
nu provisioning/tests/config_validation_tests.nu
provisioning --check server list
```

### Management
```bash
provisioning workspace info
provisioning workspace list
provisioning workspace activate production-cluster
```

## Success Metrics

- ✅ **Migration**: Completed without errors
- ✅ **Validation**: All configurations valid
- ✅ **Tests**: 6/6 tests passed
- ✅ **Operations**: Check mode successful
- ✅ **CI/CD**: Pipeline updated and working
- ✅ **Documentation**: Team updated

## Lessons Learned

1. **Always dry-run first** - Caught potential path conflicts
2. **Backup is essential** - Easy rollback if needed
3. **Validate early and often** - Caught configuration errors immediately
4. **Test in check mode** - Verified operations before deployment
5. **Document customizations** - Team knows what changed

## Next Steps

1. ✅ Monitor first deployment with new configuration
2. ✅ Train team on new workspace structure
3. ✅ Update runbooks and documentation
4. ✅ Create additional workspaces for staging/dev
5. ✅ Implement workspace templates for new projects