606 lines
15 KiB
Markdown
606 lines
15 KiB
Markdown
# Migration Example: Complete Walkthrough
|
|
|
|
## Scenario
|
|
|
|
Migrating a production infrastructure project from the old `config.defaults.toml` system to the new workspace-based configuration.
|
|
|
|
**Current Setup**:
|
|
- Infrastructure name: `production-cluster`
|
|
- Providers: AWS, UpCloud
|
|
- Platform services: Orchestrator, Control Center, KMS
|
|
- Old config location: `/Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml`
|
|
|
|
**Target Setup**:
|
|
- Workspace name: `production-cluster`
|
|
- Workspace path: `~/workspaces/production-cluster`
|
|
- All configurations validated and tested
|
|
|
|
## Step-by-Step Migration
|
|
|
|
### Step 1: Pre-Migration Assessment
|
|
|
|
```bash
|
|
# Check current configuration
|
|
$ provisioning env
|
|
|
|
# Output:
|
|
PROVISIONING_BASE: /Users/Akasha/project-provisioning
|
|
PROVISIONING_CONFIG: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
|
|
PROVISIONING_PROVIDER: aws
|
|
...
|
|
|
|
# Review current config file
|
|
$ cat provisioning/config/config.defaults.toml | head -20
|
|
|
|
[core]
|
|
name = "provisioning"
|
|
version = "1.0.0"
|
|
|
|
[debug]
|
|
enabled = false
|
|
log_level = "info"
|
|
|
|
[providers]
|
|
default = "aws"
|
|
active = ["aws", "upcloud"]
|
|
|
|
[providers.aws]
|
|
region = "us-east-1"
|
|
...
|
|
```
|
|
|
|
### Step 2: Backup Current Configuration
|
|
|
|
```bash
|
|
# Create timestamped backup
|
|
$ cp -r provisioning/config provisioning/config.backup.$(date +%Y%m%d-%H%M%S)
|
|
|
|
# Verify backup
|
|
$ ls -la provisioning/config.backup.*
|
|
|
|
drwxr-xr-x 8 user staff 256 Oct 6 10:30 provisioning/config.backup.20251006-103000
|
|
```
|
|
|
|
### Step 3: Dry Run Migration
|
|
|
|
```bash
|
|
# Preview migration without making changes
|
|
$ ./provisioning/scripts/migrate-to-target-configs.nu \
|
|
--workspace-name "production-cluster" \
|
|
--dry-run
|
|
|
|
🔄 Migration to Target-Based Configuration System
|
|
==================================================
|
|
|
|
⚠️ DRY RUN MODE - No changes will be made
|
|
|
|
Step 1: Detecting old configuration...
|
|
Found: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
|
|
✅ Old config loaded
|
|
|
|
Step 3: Target workspace
|
|
Name: production-cluster
|
|
Path: /Users/Akasha/workspaces/production-cluster
|
|
|
|
Would create workspace structure...
|
|
|
|
Step 5: Generating provisioning.yaml...
|
|
Would write to: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
|
|
Config preview:
|
|
workspace:
|
|
name: production-cluster
|
|
version: 1.0.0
|
|
created: '2025-10-06T10:35:00Z'
|
|
paths:
|
|
base: /Users/Akasha/workspaces/production-cluster
|
|
infra: /Users/Akasha/workspaces/production-cluster/infra
|
|
cache: /Users/Akasha/workspaces/production-cluster/.cache
|
|
runtime: /Users/Akasha/workspaces/production-cluster/.runtime
|
|
core:
|
|
name: provisioning
|
|
version: 1.0.0
|
|
debug:
|
|
enabled: false
|
|
log_level: info
|
|
providers:
|
|
active:
|
|
- aws
|
|
- upcloud
|
|
default: aws
|
|
...
|
|
|
|
Step 6: Migrating provider configs...
|
|
• Migrating aws...
|
|
Would create: /Users/Akasha/workspaces/production-cluster/config/providers/aws.toml
|
|
• Migrating upcloud...
|
|
Would create: /Users/Akasha/workspaces/production-cluster/config/providers/upcloud.toml
|
|
|
|
Step 7: Creating user context...
|
|
Would create: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml
|
|
```
|
|
|
|
### Step 4: Execute Migration with Backup
|
|
|
|
```bash
|
|
# Run actual migration
|
|
$ ./provisioning/scripts/migrate-to-target-configs.nu \
|
|
--workspace-name "production-cluster" \
|
|
--backup
|
|
|
|
🔄 Migration to Target-Based Configuration System
|
|
==================================================
|
|
|
|
Step 1: Detecting old configuration...
|
|
Found: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
|
|
✅ Old config loaded
|
|
|
|
Step 2: Creating backup...
|
|
✅ Backup created: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml.backup.20251006-103500
|
|
|
|
Step 3: Target workspace
|
|
Name: production-cluster
|
|
Path: /Users/Akasha/workspaces/production-cluster
|
|
|
|
Step 4: Creating workspace...
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/config
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/config/platform
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/infra
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/.cache
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/.runtime
|
|
|
|
Step 5: Generating provisioning.yaml...
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
|
|
|
|
Step 6: Migrating provider configs...
|
|
• Migrating aws...
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers/aws.toml
|
|
• Migrating upcloud...
|
|
✅ Created: /Users/Akasha/workspaces/production-cluster/config/providers/upcloud.toml
|
|
|
|
Step 7: Creating user context...
|
|
✅ Created: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml
|
|
|
|
✅ Migration Complete!
|
|
|
|
📋 Summary:
|
|
Workspace: production-cluster
|
|
Path: /Users/Akasha/workspaces/production-cluster
|
|
Config: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
|
|
Context: /Users/Akasha/Library/Application Support/provisioning/ws_production-cluster.yaml
|
|
|
|
🎯 Next Steps:
|
|
1. Review and customize: /Users/Akasha/workspaces/production-cluster/config/provisioning.yaml
|
|
2. Configure providers in: /Users/Akasha/workspaces/production-cluster/config/providers/
|
|
3. Test: provisioning workspace config validate
|
|
4. If all good, remove old config: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
|
|
|
|
⚠️ IMPORTANT: Old config is still at /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml
|
|
Backup saved at: /Users/Akasha/project-provisioning/provisioning/config/config.defaults.toml.backup.*
|
|
Remove it manually after verifying migration
|
|
```
|
|
|
|
### Step 5: Verify Workspace Structure
|
|
|
|
```bash
|
|
# Check created workspace structure
|
|
$ tree ~/workspaces/production-cluster -L 3
|
|
|
|
/Users/Akasha/workspaces/production-cluster
|
|
├── config
|
|
│ ├── provisioning.yaml
|
|
│ ├── providers
|
|
│ │ ├── aws.toml
|
|
│ │ └── upcloud.toml
|
|
│ └── platform
|
|
├── infra
|
|
├── .cache
|
|
└── .runtime
|
|
|
|
# View main configuration
|
|
$ cat ~/workspaces/production-cluster/config/provisioning.yaml
|
|
|
|
workspace:
|
|
name: production-cluster
|
|
version: 1.0.0
|
|
created: '2025-10-06T10:35:00Z'
|
|
paths:
|
|
base: /Users/Akasha/workspaces/production-cluster
|
|
infra: /Users/Akasha/workspaces/production-cluster/infra
|
|
cache: /Users/Akasha/workspaces/production-cluster/.cache
|
|
runtime: /Users/Akasha/workspaces/production-cluster/.runtime
|
|
...
|
|
|
|
# View provider configuration
|
|
$ cat ~/workspaces/production-cluster/config/providers/aws.toml
|
|
|
|
[provider]
|
|
name = "aws"
|
|
region = "us-east-1"
|
|
enabled = true
|
|
|
|
[credentials]
|
|
type = "environment"
|
|
|
|
[compute]
|
|
default_instance_type = "t3.medium"
|
|
...
|
|
```
|
|
|
|
### Step 6: Validate Configuration
|
|
|
|
```bash
|
|
# Validate workspace configuration
|
|
$ provisioning workspace config validate
|
|
|
|
✅ Validation passed
|
|
|
|
# Validate provider configurations
|
|
$ provisioning provider validate aws
|
|
|
|
✅ Validation passed
|
|
|
|
$ provisioning provider validate upcloud
|
|
|
|
✅ Validation passed
|
|
```
|
|
|
|
### Step 7: Run Validation Test Suite
|
|
|
|
```bash
|
|
# Run comprehensive validation tests
|
|
$ nu provisioning/tests/config_validation_tests.nu
|
|
|
|
🧪 Configuration Validation Test Suite
|
|
======================================
|
|
|
|
Test 1: Required Fields Validation
|
|
✅ PASSED
|
|
|
|
Test 2: Type Validation
|
|
✅ PASSED
|
|
|
|
Test 3: Enum Validation
|
|
✅ PASSED
|
|
|
|
Test 4: Range Validation
|
|
✅ PASSED
|
|
|
|
Test 5: Pattern Validation
|
|
✅ PASSED
|
|
|
|
Test 6: Deprecated Fields Warning
|
|
✅ PASSED
|
|
|
|
📊 Results: 6 passed, 0 failed
|
|
✅ All tests passed!
|
|
```
|
|
|
|
### Step 8: Test Operations
|
|
|
|
```bash
|
|
# Test server operations in check mode
|
|
$ provisioning --check server list
|
|
|
|
Checking server configuration...
|
|
✅ Configuration valid
|
|
✅ AWS provider configured
|
|
✅ UpCloud provider configured
|
|
|
|
# Test taskserv operations
|
|
$ provisioning --check taskserv list
|
|
|
|
Available taskservs:
|
|
• kubernetes (v1.28.0)
|
|
• containerd (v1.7.0)
|
|
• etcd (v3.5.0)
|
|
...
|
|
|
|
# Verify workspace info
|
|
$ provisioning workspace info
|
|
|
|
Workspace: production-cluster
|
|
Path: /Users/Akasha/workspaces/production-cluster
|
|
Active: Yes
|
|
Version: 1.0.0
|
|
Created: 2025-10-06T10:35:00Z
|
|
|
|
Configuration:
|
|
Providers: aws, upcloud
|
|
Platform services: orchestrator, control-center, kms
|
|
|
|
Infrastructure:
|
|
Count: 0
|
|
Path: /Users/Akasha/workspaces/production-cluster/infra
|
|
```
|
|
|
|
### Step 9: Customize Configuration
|
|
|
|
```bash
|
|
# Edit main workspace config
|
|
$ vim ~/workspaces/production-cluster/config/provisioning.yaml
|
|
|
|
# Update debug settings for production
|
|
debug:
|
|
enabled: false # ← Ensure disabled in production
|
|
log_level: "warn" # ← Higher threshold
|
|
|
|
# Update output settings
|
|
output:
|
|
format: "json" # ← JSON for automation
|
|
file_viewer: "jq" # ← Better for JSON
|
|
|
|
# Save and validate
|
|
$ provisioning workspace config validate
|
|
|
|
✅ Validation passed
|
|
|
|
# Edit AWS provider config
|
|
$ vim ~/workspaces/production-cluster/config/providers/aws.toml
|
|
|
|
# Update region and credentials
|
|
[provider]
|
|
region = "eu-west-1" # ← European region
|
|
|
|
[credentials]
|
|
type = "iam_role" # ← Use IAM role in production
|
|
|
|
# Validate provider config
|
|
$ provisioning provider validate aws
|
|
|
|
✅ Validation passed
|
|
```
|
|
|
|
### Step 10: Create Platform Service Configs
|
|
|
|
```bash
|
|
# Create orchestrator config
|
|
$ cat > ~/workspaces/production-cluster/config/platform/orchestrator.toml << 'EOF'
|
|
[service]
|
|
name = "orchestrator"
|
|
enabled = true
|
|
|
|
[server]
|
|
host = "0.0.0.0"
|
|
port = 8080
|
|
|
|
[workers]
|
|
count = 4
|
|
|
|
[queue]
|
|
max_size = 1000
|
|
storage_path = "/Users/Akasha/workspaces/production-cluster/.runtime/queue"
|
|
EOF
|
|
|
|
# Validate orchestrator config
|
|
$ provisioning platform validate orchestrator
|
|
|
|
✅ Validation passed
|
|
|
|
# Create KMS config
|
|
$ cat > ~/workspaces/production-cluster/config/platform/kms.toml << 'EOF'
|
|
[kms]
|
|
enabled = true
|
|
provider = "aws_kms"
|
|
|
|
[encryption]
|
|
algorithm = "AES-256-GCM"
|
|
key_rotation_days = 90
|
|
EOF
|
|
|
|
# Validate KMS config
|
|
$ provisioning platform validate kms
|
|
|
|
✅ Validation passed
|
|
```
|
|
|
|
### Step 11: Update Infrastructure Definitions
|
|
|
|
```bash
|
|
# Copy existing infrastructure to new workspace
|
|
$ cp -r provisioning/infra/production-cluster \
|
|
~/workspaces/production-cluster/infra/
|
|
|
|
# Verify infrastructure
|
|
$ ls -la ~/workspaces/production-cluster/infra/production-cluster
|
|
|
|
total 32
|
|
-rw-r--r-- 1 user staff 1024 Oct 6 10:45 servers.yaml
|
|
-rw-r--r-- 1 user staff 2048 Oct 6 10:45 taskservs.yaml
|
|
-rw-r--r-- 1 user staff 1536 Oct 6 10:45 cluster.yaml
|
|
```
|
|
|
|
### Step 12: Final Validation
|
|
|
|
```bash
|
|
# Comprehensive validation
|
|
$ provisioning workspace config validate --verbose
|
|
|
|
Validating workspace configuration...
|
|
✅ Workspace name: production-cluster (valid)
|
|
✅ Version: 1.0.0 (valid semver)
|
|
✅ Paths configuration: valid
|
|
✅ Debug configuration: valid
|
|
✅ Output configuration: valid
|
|
✅ Provider configuration: valid
|
|
✅ Secrets configuration: valid
|
|
|
|
Validating providers...
|
|
✅ AWS provider: configured and valid
|
|
✅ UpCloud provider: configured and valid
|
|
|
|
Validating platform services...
|
|
✅ Orchestrator: configured and valid
|
|
✅ KMS: configured and valid
|
|
|
|
Overall: ✅ All validations passed
|
|
```
|
|
|
|
### Step 13: Test Deployment (Check Mode)
|
|
|
|
```bash
|
|
# Test server deployment in check mode
|
|
$ provisioning --check server create --infra production-cluster
|
|
|
|
Checking server deployment...
|
|
✅ Configuration loaded
|
|
✅ Provider validated (aws)
|
|
✅ Infrastructure definition found
|
|
✅ Resource limits checked
|
|
✅ Dependencies resolved
|
|
|
|
Would create:
|
|
• web-01 (t3.medium, eu-west-1a)
|
|
• web-02 (t3.medium, eu-west-1b)
|
|
• db-01 (r5.large, eu-west-1a)
|
|
|
|
Dry run complete. Use without --check to deploy.
|
|
```
|
|
|
|
### Step 14: Clean Up Old Configuration
|
|
|
|
```bash
|
|
# Verify everything works with new configuration
|
|
$ provisioning workspace info
|
|
$ provisioning --check server list
|
|
$ provisioning --check taskserv list
|
|
|
|
# All good? Remove old config
|
|
$ rm provisioning/config/config.defaults.toml
|
|
$ rm provisioning/config/config.user.toml
|
|
|
|
# Keep backup for reference
|
|
$ ls -la provisioning/config.backup.*
|
|
|
|
-rw-r--r-- 1 user staff 8192 Oct 6 10:30 config.defaults.toml.backup.20251006-103500
|
|
```
|
|
|
|
### Step 15: Update CI/CD Pipeline
|
|
|
|
```yaml
|
|
# .gitlab-ci.yml
|
|
variables:
|
|
PROVISIONING_WORKSPACE: "production-cluster"
|
|
|
|
stages:
|
|
- validate
|
|
- deploy
|
|
|
|
validate:
|
|
stage: validate
|
|
script:
|
|
# Validate all configurations
|
|
- provisioning workspace config validate
|
|
- provisioning provider validate --all
|
|
- provisioning platform validate --all
|
|
# Run validation tests
|
|
- nu provisioning/tests/config_validation_tests.nu
|
|
only:
|
|
changes:
|
|
- "workspaces/*/config/**/*"
|
|
|
|
deploy:
|
|
stage: deploy
|
|
script:
|
|
# Deploy with validated configuration
|
|
- provisioning --check server create --infra production-cluster
|
|
- provisioning server create --infra production-cluster
|
|
only:
|
|
- main
|
|
when: manual
|
|
```
|
|
|
|
## Result
|
|
|
|
**Before Migration**:
|
|
- ❌ Monolithic config file
|
|
- ❌ No validation
|
|
- ❌ Hard to manage multiple environments
|
|
- ❌ Provider configs mixed with core settings
|
|
|
|
**After Migration**:
|
|
- ✅ Modular workspace structure
|
|
- ✅ Schema-based validation
|
|
- ✅ Clear separation of concerns
|
|
- ✅ Provider configs isolated
|
|
- ✅ Platform services configurable
|
|
- ✅ Test suite for validation
|
|
- ✅ CI/CD integration ready
|
|
|
|
## Workspace Final State
|
|
|
|
```
|
|
~/workspaces/production-cluster/
|
|
├── config/
|
|
│ ├── provisioning.yaml # ✅ Main config (validated)
|
|
│ ├── providers/
|
|
│ │ ├── aws.toml # ✅ AWS config (validated)
|
|
│ │ └── upcloud.toml # ✅ UpCloud config (validated)
|
|
│ └── platform/
|
|
│ ├── orchestrator.toml # ✅ Orchestrator (validated)
|
|
│ └── kms.toml # ✅ KMS (validated)
|
|
├── infra/
|
|
│ └── production-cluster/ # ✅ Infrastructure definitions
|
|
│ ├── servers.yaml
|
|
│ ├── taskservs.yaml
|
|
│ └── cluster.yaml
|
|
├── .cache/ # ✅ Cache directory
|
|
└── .runtime/ # ✅ Runtime data
|
|
|
|
~/Library/Application Support/provisioning/
|
|
└── ws_production-cluster.yaml # ✅ User context
|
|
```
|
|
|
|
## Commands Reference
|
|
|
|
### Migration
|
|
```bash
|
|
./provisioning/scripts/migrate-to-target-configs.nu --workspace-name "production-cluster" --backup
|
|
```
|
|
|
|
### Validation
|
|
```bash
|
|
provisioning workspace config validate
|
|
provisioning provider validate aws
|
|
provisioning platform validate orchestrator
|
|
```
|
|
|
|
### Testing
|
|
```bash
|
|
nu provisioning/tests/config_validation_tests.nu
|
|
provisioning --check server list
|
|
```
|
|
|
|
### Management
|
|
```bash
|
|
provisioning workspace info
|
|
provisioning workspace list
|
|
provisioning workspace activate production-cluster
|
|
```
|
|
|
|
## Success Metrics
|
|
|
|
- ✅ **Migration**: Completed without errors
|
|
- ✅ **Validation**: All configurations valid
|
|
- ✅ **Tests**: 6/6 tests passed
|
|
- ✅ **Operations**: Check mode successful
|
|
- ✅ **CI/CD**: Pipeline updated and working
|
|
- ✅ **Documentation**: Team updated
|
|
|
|
## Lessons Learned
|
|
|
|
1. **Always dry-run first** - Caught potential path conflicts
|
|
2. **Backup is essential** - Easy rollback if needed
|
|
3. **Validate early and often** - Caught configuration errors immediately
|
|
4. **Test in check mode** - Verified operations before deployment
|
|
5. **Document customizations** - Team knows what changed
|
|
|
|
## Next Steps
|
|
|
|
1. ✅ Monitor first deployment with new configuration
|
|
2. ✅ Train team on new workspace structure
|
|
3. ✅ Update runbooks and documentation
|
|
4. ✅ Create additional workspaces for staging/dev
|
|
5. ✅ Implement workspace templates for new projects
|