jesus/provisioning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
provisioning/docs/MIGRATION_EXAMPLE.md
Jesús Pérez 897f9d8ed1
chore update projects
2025-10-07 11:12:02 +01:00

15 KiB
Raw Blame History

Migration Example: Complete Walkthrough

Scenario

Migrating a production infrastructure project from the old config.defaults.toml system to the new workspace-based configuration.

Current Setup:

  • Infrastructure name: production-cluster
  • Providers: AWS, UpCloud
  • Platform services: Orchestrator, Control Center, KMS
  • Old config location: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml

Target Setup:

  • Workspace name: production-cluster
  • Workspace path: ~/workspaces/production-cluster
  • All configurations validated and tested

Step-by-Step Migration

Step 1: Pre-Migration Assessment

# Check current configuration
$ provisioning env

# Output:
PROVISIONING_BASE: /Users/Akasha/project-provisioning
PROVISIONING_CONFIG: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
PROVISIONING_PROVIDER: aws
...

# Review current config file
$ cat provisioning/config/config.defaults.toml | head -20

[core]
name = "provisioning"
version = "1.0.0"

[debug]
enabled = false
log_level = "info"

[providers]
default = "aws"
active = ["aws", "upcloud"]

[providers.aws]
region = "us-east-1"
...

Step 2: Backup Current Configuration

# Create timestamped backup
$ cp -r provisioning/config provisioning/config.backup.$(date +%Y%m%d-%H%M%S)

# Verify backup
$ ls -la provisioning/config.backup.*

drwxr-xr-x  8 user  staff  256 Oct  6 10:30 provisioning/config.backup.20251006-103000

Step 3: Dry Run Migration

# Preview migration without making changes
$ ./provisioning/scripts/migrate-to-target-configs.nu \
    --workspace-name "production-cluster" \
    --dry-run

🔄 Migration to Target-Based Configuration System
==================================================

⚠️  DRY RUN MODE - No changes will be made

Step 1: Detecting old configuration...
   Found: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
   ✅ Old config loaded

Step 3: Target workspace
   Name: production-cluster
   Path: /Users/Akasha/workspaces/production-cluster

   Would create workspace structure...

Step 5: Generating provisioning.yaml...
   Would write to: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
   Config preview:
workspace:
  name: production-cluster
  version: 1.0.0
  created: '2025-10-06T10:35:00Z'
paths:
  base: /Users/Akasha/workspaces/production-cluster
  infra: /Users/Akasha/workspaces/production-cluster/infra
  cache: /Users/Akasha/workspaces/production-cluster/.cache
  runtime: /Users/Akasha/workspaces/production-cluster/.runtime
core:
  name: provisioning
  version: 1.0.0
debug:
  enabled: false
  log_level: info
providers:
  active:
  - aws
  - upcloud
  default: aws
...

Step 6: Migrating provider configs...
   • Migrating aws...
     Would create: /Users/Akasha/workspaces/production-cluster/config/providers/aws.toml
   • Migrating upcloud...
     Would create: /Users/Akasha/workspaces/production-cluster/config/providers/upcloud.toml

Step 7: Creating user context...
   Would create: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml

Step 4: Execute Migration with Backup

# Run actual migration
$ ./provisioning/scripts/migrate-to-target-configs.nu \
    --workspace-name "production-cluster" \
    --backup

🔄 Migration to Target-Based Configuration System
==================================================

Step 1: Detecting old configuration...
   Found: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
   ✅ Old config loaded

Step 2: Creating backup...
   ✅ Backup created: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml.backup.20251006-103500

Step 3: Target workspace
   Name: production-cluster
   Path: /Users/Akasha/workspaces/production-cluster

Step 4: Creating workspace...
   ✅ Created: /Users/Akasha/workspaces/production-cluster
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config/platform
   ✅ Created: /Users/Akasha/workspaces/production-cluster/infra
   ✅ Created: /Users/Akasha/workspaces/production-cluster/.cache
   ✅ Created: /Users/Akasha/workspaces/production-cluster/.runtime

Step 5: Generating provisioning.yaml...
   ✅ Created: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml

Step 6: Migrating provider configs...
   • Migrating aws...
     ✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers/aws.toml
   • Migrating upcloud...
     ✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers/upcloud.toml

Step 7: Creating user context...
   ✅ Created: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml

✅ Migration Complete!

📋 Summary:
   Workspace: production-cluster
   Path: /Users/Akasha/workspaces/production-cluster
   Config: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
   Context: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml

🎯 Next Steps:
   1. Review and customize: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
   2. Configure providers in: /Users/Akasha/workspaces/production-cluster/config/providers/
   3. Test: provisioning workspace config validate
   4. If all good, remove old config: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml

⚠️  IMPORTANT: Old config is still at /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
   Backup saved at: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml.backup.*
   Remove it manually after verifying migration

Step 5: Verify Workspace Structure

# Check created workspace structure
$ tree ~/workspaces/production-cluster -L 3

/Users/Akasha/workspaces/production-cluster
├── config
│   ├── provisioning.yaml
│   ├── providers
│   │   ├── aws.toml
│   │   └── upcloud.toml
│   └── platform
├── infra
├── .cache
└── .runtime

# View main configuration
$ cat ~/workspaces/production-cluster/config/provisioning.yaml

workspace:
  name: production-cluster
  version: 1.0.0
  created: '2025-10-06T10:35:00Z'
paths:
  base: /Users/Akasha/workspaces/production-cluster
  infra: /Users/Akasha/workspaces/production-cluster/infra
  cache: /Users/Akasha/workspaces/production-cluster/.cache
  runtime: /Users/Akasha/workspaces/production-cluster/.runtime
...

# View provider configuration
$ cat ~/workspaces/production-cluster/config/providers/aws.toml

[provider]
name = "aws"
region = "us-east-1"
enabled = true

[credentials]
type = "environment"

[compute]
default_instance_type = "t3.medium"
...

Step 6: Validate Configuration

# Validate workspace configuration
$ provisioning workspace config validate

✅ Validation passed

# Validate provider configurations
$ provisioning provider validate aws

✅ Validation passed

$ provisioning provider validate upcloud

✅ Validation passed

Step 7: Run Validation Test Suite

# Run comprehensive validation tests
$ nu provisioning/tests/config_validation_tests.nu

🧪 Configuration Validation Test Suite
======================================

Test 1: Required Fields Validation
  ✅ PASSED

Test 2: Type Validation
  ✅ PASSED

Test 3: Enum Validation
  ✅ PASSED

Test 4: Range Validation
  ✅ PASSED

Test 5: Pattern Validation
  ✅ PASSED

Test 6: Deprecated Fields Warning
  ✅ PASSED

📊 Results: 6 passed, 0 failed
✅ All tests passed!

Step 8: Test Operations

# Test server operations in check mode
$ provisioning --check server list

Checking server configuration...
✅ Configuration valid
✅ AWS provider configured
✅ UpCloud provider configured

# Test taskserv operations
$ provisioning --check taskserv list

Available taskservs:
  • kubernetes (v1.28.0)
  • containerd (v1.7.0)
  • etcd (v3.5.0)
  ...

# Verify workspace info
$ provisioning workspace info

Workspace: production-cluster
Path: /Users/Akasha/workspaces/production-cluster
Active: Yes
Version: 1.0.0
Created: 2025-10-06T10:35:00Z

Configuration:
  Providers: aws, upcloud
  Platform services: orchestrator, control-center, kms

Infrastructure:
  Count: 0
  Path: /Users/Akasha/workspaces/production-cluster/infra

Step 9: Customize Configuration

# Edit main workspace config
$ vim ~/workspaces/production-cluster/config/provisioning.yaml

# Update debug settings for production
debug:
  enabled: false  # ← Ensure disabled in production
  log_level: "warn"  # ← Higher threshold

# Update output settings
output:
  format: "json"  # ← JSON for automation
  file_viewer: "jq"  # ← Better for JSON

# Save and validate
$ provisioning workspace config validate

✅ Validation passed

# Edit AWS provider config
$ vim ~/workspaces/production-cluster/config/providers/aws.toml

# Update region and credentials
[provider]
region = "eu-west-1"  # ← European region

[credentials]
type = "iam_role"  # ← Use IAM role in production

# Validate provider config
$ provisioning provider validate aws

✅ Validation passed

Step 10: Create Platform Service Configs

# Create orchestrator config
$ cat > ~/workspaces/production-cluster/config/platform/orchestrator.toml << 'EOF'
[service]
name = "orchestrator"
enabled = true

[server]
host = "0.0.0.0"
port = 8080

[workers]
count = 4

[queue]
max_size = 1000
storage_path = "/Users/Akasha/workspaces/production-cluster/.runtime/queue"
EOF

# Validate orchestrator config
$ provisioning platform validate orchestrator

✅ Validation passed

# Create KMS config
$ cat > ~/workspaces/production-cluster/config/platform/kms.toml << 'EOF'
[kms]
enabled = true
provider = "aws_kms"

[encryption]
algorithm = "AES-256-GCM"
key_rotation_days = 90
EOF

# Validate KMS config
$ provisioning platform validate kms

✅ Validation passed

Step 11: Update Infrastructure Definitions

# Copy existing infrastructure to new workspace
$ cp -r provisioning/infra/production-cluster \
       ~/workspaces/production-cluster/infra/

# Verify infrastructure
$ ls -la ~/workspaces/production-cluster/infra/production-cluster

total 32
-rw-r--r--  1 user  staff  1024 Oct  6 10:45 servers.yaml
-rw-r--r--  1 user  staff  2048 Oct  6 10:45 taskservs.yaml
-rw-r--r--  1 user  staff  1536 Oct  6 10:45 cluster.yaml

Step 12: Final Validation

# Comprehensive validation
$ provisioning workspace config validate --verbose

Validating workspace configuration...
✅ Workspace name: production-cluster (valid)
✅ Version: 1.0.0 (valid semver)
✅ Paths configuration: valid
✅ Debug configuration: valid
✅ Output configuration: valid
✅ Provider configuration: valid
✅ Secrets configuration: valid

Validating providers...
✅ AWS provider: configured and valid
✅ UpCloud provider: configured and valid

Validating platform services...
✅ Orchestrator: configured and valid
✅ KMS: configured and valid

Overall: ✅ All validations passed

Step 13: Test Deployment (Check Mode)

# Test server deployment in check mode
$ provisioning --check server create --infra production-cluster

Checking server deployment...
✅ Configuration loaded
✅ Provider validated (aws)
✅ Infrastructure definition found
✅ Resource limits checked
✅ Dependencies resolved

Would create:
  • web-01 (t3.medium, eu-west-1a)
  • web-02 (t3.medium, eu-west-1b)
  • db-01 (r5.large, eu-west-1a)

Dry run complete. Use without --check to deploy.

Step 14: Clean Up Old Configuration

# Verify everything works with new configuration
$ provisioning workspace info
$ provisioning --check server list
$ provisioning --check taskserv list

# All good? Remove old config
$ rm provisioning/config/config.defaults.toml
$ rm provisioning/config/config.user.toml

# Keep backup for reference
$ ls -la provisioning/config.backup.*

-rw-r--r--  1 user  staff  8192 Oct  6 10:30 config.defaults.toml.backup.20251006-103500

Step 15: Update CI/CD Pipeline

# .gitlab-ci.yml
variables:
  PROVISIONING_WORKSPACE: "production-cluster"

stages:
  - validate
  - deploy

validate:
  stage: validate
  script:
    # Validate all configurations
    - provisioning workspace config validate
    - provisioning provider validate --all
    - provisioning platform validate --all
    # Run validation tests
    - nu provisioning/tests/config_validation_tests.nu
  only:
    changes:
      - "workspaces/*/config/**/*"

deploy:
  stage: deploy
  script:
    # Deploy with validated configuration
    - provisioning --check server create --infra production-cluster
    - provisioning server create --infra production-cluster
  only:
    - main
  when: manual

Result

Before Migration:

  • ❌ Monolithic config file
  • ❌ No validation
  • ❌ Hard to manage multiple environments
  • ❌ Provider configs mixed with core settings

After Migration:

  • ✅ Modular workspace structure
  • ✅ Schema-based validation
  • ✅ Clear separation of concerns
  • ✅ Provider configs isolated
  • ✅ Platform services configurable
  • ✅ Test suite for validation
  • ✅ CI/CD integration ready

Workspace Final State

~/workspaces/production-cluster/
├── config/
│   ├── provisioning.yaml          # ✅ Main config (validated)
│   ├── providers/
│   │   ├── aws.toml               # ✅ AWS config (validated)
│   │   └── upcloud.toml           # ✅ UpCloud config (validated)
│   └── platform/
│       ├── orchestrator.toml      # ✅ Orchestrator (validated)
│       └── kms.toml               # ✅ KMS (validated)
├── infra/
│   └── production-cluster/        # ✅ Infrastructure definitions
│       ├── servers.yaml
│       ├── taskservs.yaml
│       └── cluster.yaml
├── .cache/                        # ✅ Cache directory
└── .runtime/                      # ✅ Runtime data

~/Library/Application Support/provisioning/
└── ws_production-cluster.yaml     # ✅ User context

Commands Reference

Migration

./provisioning/scripts/migrate-to-target-configs.nu --workspace-name "production-cluster" --backup

Validation

provisioning workspace config validate
provisioning provider validate aws
provisioning platform validate orchestrator

Testing

nu provisioning/tests/config_validation_tests.nu
provisioning --check server list

Management

provisioning workspace info
provisioning workspace list
provisioning workspace activate production-cluster

Success Metrics

  • ✅ Migration: Completed without errors
  • ✅ Validation: All configurations valid
  • ✅ Tests: 6/6 tests passed
  • ✅ Operations: Check mode successful
  • ✅ CI/CD: Pipeline updated and working
  • ✅ Documentation: Team updated

Lessons Learned

  1. Always dry-run first - Caught potential path conflicts
  2. Backup is essential - Easy rollback if needed
  3. Validate early and often - Caught configuration errors immediately
  4. Test in check mode - Verified operations before deployment
  5. Document customizations - Team knows what changed

Next Steps

  1. ✅ Monitor first deployment with new configuration
  2. ✅ Train team on new workspace structure
  3. ✅ Update runbooks and documentation
  4. ✅ Create additional workspaces for staging/dev
  5. ✅ Implement workspace templates for new projects