` #### Token Expired ```plaintext\\nError: Token verification failed\\n```plaintext **Fix**: Re-login: `provisioning auth login ` --- ### Troubleshooting | Error | Solution |\\n|-------|----------|\\n| Plugin not available | `plugin add target/release/nu_plugin_auth` |\\n| Control center offline | Start: `cd provisioning/platform/control-center && cargo run` |\\n| Invalid MFA code | Get fresh code (expires in 30s) |\\n| Token expired | Re-login: `provisioning auth login ` |\\n| Keyring access denied | Grant app access in system settings | --- ### Audit Logs ```bash\\n# View audit log\\ncat provisioning/logs/audit.log # Filter by user\\ncat provisioning/logs/audit.log | jq \'. | select(.user == \\"admin\\")\' # Filter by operation\\ncat provisioning/logs/audit.log | jq \'. | select(.operation == \\"server_create\\")\'\\n```plaintext --- ### CI/CD Integration #### Option 1: Skip Auth (Dev/Test Only) ```bash\\nexport PROVISIONING_SKIP_AUTH=true\\nprovisioning server create ci-server\\n```plaintext #### Option 2: Check Mode ```bash\\nprovisioning server create ci-server --check\\n```plaintext #### Option 3: Service Account (Future) ```bash\\nexport PROVISIONING_AUTH_TOKEN=\\"\\"\\nprovisioning server create ci-server\\n```plaintext --- ### Performance | Operation | Auth Overhead |\\n|-----------|---------------|\\n| Server create | ~20ms |\\n| Taskserv create | ~20ms |\\n| Batch submit | ~20ms |\\n| Check mode | 0ms (skipped) | --- ### Related Docs - **Full Guide**: `docs/user/AUTHENTICATION_LAYER_GUIDE.md`\\n- **Implementation**: `AUTHENTICATION_LAYER_IMPLEMENTATION_SUMMARY.md`\\n- **Security ADR**: `docs/architecture/ADR-009-security-system-complete.md` --- **Quick Help**: `provisioning help auth` or `provisioning auth --help` --- **Last Updated**: 2025-10-09\\n**Maintained By**: Security Team --- ## Setup Guide ### Complete Authentication Setup Guide Current Settings (from your config) ```plaintext\\n[security]\\nrequire_auth = true # ✅ Auth is REQUIRED\\nallow_skip_auth = false # ❌ Cannot skip with env var\\nauth_timeout = 3600 # Token valid for 1 hour [platform.control_center]\\nurl = \\"http://localhost:3000\\" # Control Center endpoint\\n```plaintext ### STEP 1: Start Control Center The Control Center is the authentication backend: ```bash\\n# Check if it\'s already running\\ncurl http://localhost:3000/health # If not running, start it\\ncd /Users/Akasha/project-provisioning/provisioning/platform/control-center\\ncargo run --release & # Wait for it to start (may take 30-60 seconds)\\nsleep 30\\ncurl http://localhost:3000/health\\n```plaintext Expected Output: ```json\\n{\\"status\\": \\"healthy\\"}\\n```plaintext ### STEP 2: Find Default Credentials Check for default user setup: ```bash\\n# Look for initialization scripts\\nls -la /Users/Akasha/project-provisioning/provisioning/platform/control-center/ # Check for README or setup instructions\\ncat /Users/Akasha/project-provisioning/provisioning/platform/control-center/README.md # Or check for default config\\ncat /Users/Akasha/project-provisioning/provisioning/platform/control-center/config.toml 2>/dev/null || echo \\"Config not found\\"\\n```plaintext ### STEP 3: Log In Once you have credentials (usually admin / password from setup): ```bash\\n# Interactive login - will prompt for password\\nprovisioning auth login # Or with username\\nprovisioning auth login admin # Verify you\'re logged in\\nprovisioning auth status\\n```plaintext Expected Success Output: ```plaintext\\n✓ Login successful! User: admin\\nRole: admin\\nExpires: 2025-10-22T14:30:00Z\\nMFA: false Session active and ready\\n```plaintext ### STEP 4: Now Create Your Server Once authenticated: ```bash\\n# Try server creation again\\nprovisioning server create sgoyol --check # Or with full details\\nprovisioning server create sgoyol --infra workspace_librecloud --check\\n```plaintext ### 🛠️ Alternative: Skip Auth for Development If you want to bypass authentication temporarily for testing: #### Option A: Edit config to allow skip ```bash\\n# You would need to parse and modify TOML - easier to do next option\\n```plaintext #### Option B: Use environment variable (if allowed by config) ```bash\\nexport PROVISIONING_SKIP_AUTH=true\\nprovisioning server create sgoyol\\nunset PROVISIONING_SKIP_AUTH\\n```plaintext #### Option C: Use check mode (always works, no auth needed) ```bash\\nprovisioning server create sgoyol --check\\n```plaintext #### Option D: Modify config.defaults.toml (permanent for dev) Edit: `provisioning/config/config.defaults.toml` Change line 193 to: ```toml\\nallow_skip_auth = true\\n```plaintext ### 🔍 Troubleshooting | Problem | Solution |\\n|----------------------------|---------------------------------------------------------------------|\\n| Control Center won\'t start | Check port 3000 not in use: `lsof -i :3000` |\\n| \\"No token found\\" error | Login with: `provisioning auth login` |\\n| Login fails | Verify Control Center is running: `curl http://localhost:3000/health` |\\n| Token expired | Re-login: `provisioning auth login` |\\n| Plugin not available | Using HTTP fallback - this is OK, works without plugin |","breadcrumbs":"Authentication Layer Guide » 1. Login to Platform","id":"1520","title":"1. Login to Platform"},"1521":{"body":"Version : 1.0.0 Last Updated : 2025-10-08 Status : Production Ready","breadcrumbs":"Config Encryption Guide » Configuration Encryption Guide","id":"1521","title":"Configuration Encryption Guide"},"1522":{"body":"The Provisioning Platform includes a comprehensive configuration encryption system that provides: Transparent Encryption/Decryption : Configs are automatically decrypted on load Multiple KMS Backends : Age, AWS KMS, HashiCorp Vault, Cosmian KMS Memory-Only Decryption : Secrets never written to disk in plaintext SOPS Integration : Industry-standard encryption with SOPS Sensitive Data Detection : Automatic scanning for unencrypted sensitive data","breadcrumbs":"Config Encryption Guide » Overview","id":"1522","title":"Overview"},"1523":{"body":"Prerequisites Quick Start Configuration Encryption KMS Backends CLI Commands Integration with Config Loader Best Practices Troubleshooting","breadcrumbs":"Config Encryption Guide » Table of Contents","id":"1523","title":"Table of Contents"},"1524":{"body":"","breadcrumbs":"Config Encryption Guide » Prerequisites","id":"1524","title":"Prerequisites"},"1525":{"body":"SOPS (v3.10.2+) # macOS\\nbrew install sops # Linux\\nwget https://github.com/mozilla/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64\\nsudo mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops\\nsudo chmod +x /usr/local/bin/sops Age (for Age backend - recommended) # macOS\\nbrew install age # Linux\\napt install age AWS CLI (for AWS KMS backend - optional) brew install awscli","breadcrumbs":"Config Encryption Guide » Required Tools","id":"1525","title":"Required Tools"},"1526":{"body":"# Check SOPS\\nsops --version # Check Age\\nage --version # Check AWS CLI (optional)\\naws --version\\n```plaintext --- ## Quick Start ### 1. Initialize Encryption Generate Age keys and create SOPS configuration: ```bash\\nprovisioning config init-encryption --kms age\\n```plaintext This will: - Generate Age key pair in `~/.config/sops/age/keys.txt`\\n- Display your public key (recipient)\\n- Create `.sops.yaml` in your project ### 2. Set Environment Variables Add to your shell profile (`~/.zshrc` or `~/.bashrc`): ```bash\\n# Age encryption\\nexport SOPS_AGE_RECIPIENTS=\\"age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p\\"\\nexport PROVISIONING_KAGE=\\"$HOME/.config/sops/age/keys.txt\\"\\n```plaintext Replace the recipient with your actual public key. ### 3. Validate Setup ```bash\\nprovisioning config validate-encryption\\n```plaintext Expected output: ```plaintext\\n✅ Encryption configuration is valid SOPS installed: true Age backend: true KMS enabled: false Errors: 0 Warnings: 0\\n```plaintext ### 4. Encrypt Your First Config ```bash\\n# Create a config with sensitive data\\ncat > workspace/config/secure.yaml < edit -> re-encrypt)\\nprovisioning config edit-secure workspace/config/secure.enc.yaml\\n```plaintext This will: 1. Decrypt the file temporarily\\n2. Open in your `$EDITOR` (vim/nano/etc)\\n3. Re-encrypt when you save and close\\n4. Remove temporary decrypted file ### Check Encryption Status ```bash\\n# Check if file is encrypted\\nprovisioning config is-encrypted workspace/config/secure.yaml # Get detailed encryption info\\nprovisioning config encryption-info workspace/config/secure.yaml\\n```plaintext --- ## KMS Backends ### Age (Recommended for Development) **Pros**: - Simple file-based keys\\n- No external dependencies\\n- Fast and secure\\n- Works offline **Setup**: ```bash\\n# Initialize\\nprovisioning config init-encryption --kms age # Set environment variables\\nexport SOPS_AGE_RECIPIENTS=\\"age1...\\" # Your public key\\nexport PROVISIONING_KAGE=\\"$HOME/.config/sops/age/keys.txt\\"\\n```plaintext **Encrypt/Decrypt**: ```bash\\nprovisioning config encrypt secrets.yaml --kms age\\nprovisioning config decrypt secrets.enc.yaml\\n```plaintext ### AWS KMS (Production) **Pros**: - Centralized key management\\n- Audit logging\\n- IAM integration\\n- Key rotation **Setup**: 1. Create KMS key in AWS Console\\n2. Configure AWS credentials: ```bash aws configure Update .sops.yaml: creation_rules: - path_regex: .*\\\\.enc\\\\.yaml$ kms: \\"arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012\\" Encrypt/Decrypt : provisioning config encrypt secrets.yaml --kms aws-kms\\nprovisioning config decrypt secrets.enc.yaml\\n```plaintext ### HashiCorp Vault (Enterprise) **Pros**: - Dynamic secrets\\n- Centralized secret management\\n- Audit logging\\n- Policy-based access **Setup**: 1. Configure Vault address and token: ```bash export VAULT_ADDR=\\"https://vault.example.com:8200\\" export VAULT_TOKEN=\\"s.xxxxxxxxxxxxxx\\" Update configuration: # workspace/config/provisioning.yaml\\nkms: enabled: true mode: \\"remote\\" vault: address: \\"https://vault.example.com:8200\\" transit_key: \\"provisioning\\" Encrypt/Decrypt : provisioning config encrypt secrets.yaml --kms vault\\nprovisioning config decrypt secrets.enc.yaml\\n```plaintext ### Cosmian KMS (Confidential Computing) **Pros**: - Confidential computing support\\n- Zero-knowledge architecture\\n- Post-quantum ready\\n- Cloud-agnostic **Setup**: 1. Deploy Cosmian KMS server\\n2. Update configuration: ```toml kms: enabled: true mode: \\"remote\\" remote: endpoint: \\"https://kms.example.com:9998\\" auth_method: \\"certificate\\" client_cert: \\"/path/to/client.crt\\" client_key: \\"/path/to/client.key\\" Encrypt/Decrypt : provisioning config encrypt secrets.yaml --kms cosmian\\nprovisioning config decrypt secrets.enc.yaml\\n```plaintext --- ## CLI Commands ### Configuration Encryption Commands | Command | Description |\\n|---------|-------------|\\n| `config encrypt ` | Encrypt configuration file |\\n| `config decrypt ` | Decrypt configuration file |\\n| `config edit-secure ` | Edit encrypted file securely |\\n| `config rotate-keys  ` | Rotate encryption keys |\\n| `config is-encrypted ` | Check if file is encrypted |\\n| `config encryption-info ` | Show encryption details |\\n| `config validate-encryption` | Validate encryption setup |\\n| `config scan-sensitive ` | Find unencrypted sensitive configs |\\n| `config encrypt-all ` | Encrypt all sensitive configs |\\n| `config init-encryption` | Initialize encryption (generate keys) | ### Examples ```bash\\n# Encrypt workspace config\\nprovisioning config encrypt workspace/config/secure.yaml --in-place # Edit encrypted file\\nprovisioning config edit-secure workspace/config/secure.yaml # Scan for unencrypted sensitive configs\\nprovisioning config scan-sensitive workspace/config --recursive # Encrypt all sensitive configs in workspace\\nprovisioning config encrypt-all workspace/config --kms age --recursive # Check encryption status\\nprovisioning config is-encrypted workspace/config/secure.yaml # Get detailed info\\nprovisioning config encryption-info workspace/config/secure.yaml # Validate setup\\nprovisioning config validate-encryption\\n```plaintext --- ## Integration with Config Loader ### Automatic Decryption The config loader automatically detects and decrypts encrypted files: ```nushell\\n# Load encrypted config (automatically decrypted in memory)\\nuse lib_provisioning/config/loader.nu let config = (load-provisioning-config --debug)\\n```plaintext **Key Features**: - **Transparent**: No code changes needed\\n- **Memory-Only**: Decrypted content never written to disk\\n- **Fallback**: If decryption fails, attempts to load as plain file\\n- **Debug Support**: Shows decryption status with `--debug` flag ### Manual Loading ```nushell\\nuse lib_provisioning/config/encryption.nu # Load encrypted config\\nlet secure_config = (load-encrypted-config \\"workspace/config/secure.enc.yaml\\") # Memory-only decryption (no file created)\\nlet decrypted_content = (decrypt-config-memory \\"workspace/config/secure.enc.yaml\\")\\n```plaintext ### Configuration Hierarchy with Encryption The system supports encrypted files at any level: ```plaintext\\n1. workspace/{name}/config/provisioning.yaml ← Can be encrypted\\n2. workspace/{name}/config/providers/*.toml ← Can be encrypted\\n3. workspace/{name}/config/platform/*.toml ← Can be encrypted\\n4. ~/.../provisioning/ws_{name}.yaml ← Can be encrypted\\n5. Environment variables (PROVISIONING_*) ← Plain text\\n```plaintext --- ## Best Practices ### 1. Encrypt All Sensitive Data **Always encrypt configs containing**: - Passwords\\n- API keys\\n- Secret keys\\n- Private keys\\n- Tokens\\n- Credentials **Scan for unencrypted sensitive data**: ```bash\\nprovisioning config scan-sensitive workspace --recursive\\n```plaintext ### 2. Use Appropriate KMS Backend | Environment | Recommended Backend |\\n|-------------|---------------------|\\n| Development | Age (file-based) |\\n| Staging | AWS KMS or Vault |\\n| Production | AWS KMS or Vault |\\n| CI/CD | AWS KMS with IAM roles | ### 3. Key Management **Age Keys**: - Store private keys securely: `~/.config/sops/age/keys.txt`\\n- Set file permissions: `chmod 600 ~/.config/sops/age/keys.txt`\\n- Backup keys securely (encrypted backup)\\n- Never commit private keys to git **AWS KMS**: - Use separate keys per environment\\n- Enable key rotation\\n- Use IAM policies for access control\\n- Monitor usage with CloudTrail **Vault**: - Use transit engine for encryption\\n- Enable audit logging\\n- Implement least-privilege policies\\n- Regular policy reviews ### 4. File Organization ```plaintext\\nworkspace/\\n└── config/ ├── provisioning.yaml # Plain (no secrets) ├── secure.yaml # Encrypted (SOPS auto-detects) ├── providers/ │ ├── aws.toml # Plain (no secrets) │ └── aws-credentials.enc.toml # Encrypted └── platform/ └── database.enc.yaml # Encrypted\\n```plaintext ### 5. Git Integration **Add to `.gitignore`**: ```gitignore\\n# Unencrypted sensitive files\\n**/secrets.yaml\\n**/credentials.yaml\\n**/*.dec.yaml\\n**/*.dec.toml # Temporary decrypted files\\n*.tmp.yaml\\n*.tmp.toml\\n```plaintext **Commit encrypted files**: ```bash\\n# Encrypted files are safe to commit\\ngit add workspace/config/secure.enc.yaml\\ngit commit -m \\"Add encrypted configuration\\"\\n```plaintext ### 6. Rotation Strategy **Regular Key Rotation**: ```bash\\n# Generate new Age key\\nage-keygen -o ~/.config/sops/age/keys-new.txt # Update .sops.yaml with new recipient # Rotate keys for file\\nprovisioning config rotate-keys workspace/config/secure.yaml \\n```plaintext **Frequency**: - Development: Annually\\n- Production: Quarterly\\n- After team member departure: Immediately ### 7. Audit and Monitoring **Track encryption status**: ```bash\\n# Regular scans\\nprovisioning config scan-sensitive workspace --recursive # Validate encryption setup\\nprovisioning config validate-encryption\\n```plaintext **Monitor access** (with Vault/AWS KMS): - Enable audit logging\\n- Review access patterns\\n- Alert on anomalies --- ## Troubleshooting ### SOPS Not Found **Error**: ```plaintext\\nSOPS binary not found\\n```plaintext **Solution**: ```bash\\n# Install SOPS\\nbrew install sops # Verify\\nsops --version\\n```plaintext ### Age Key Not Found **Error**: ```plaintext\\nAge key file not found: ~/.config/sops/age/keys.txt\\n```plaintext **Solution**: ```bash\\n# Generate new key\\nmkdir -p ~/.config/sops/age\\nage-keygen -o ~/.config/sops/age/keys.txt # Set environment variable\\nexport PROVISIONING_KAGE=\\"$HOME/.config/sops/age/keys.txt\\"\\n```plaintext ### SOPS_AGE_RECIPIENTS Not Set **Error**: ```plaintext\\nno AGE_RECIPIENTS for file.yaml\\n```plaintext **Solution**: ```bash\\n# Extract public key from private key\\ngrep \\"public key:\\" ~/.config/sops/age/keys.txt # Set environment variable\\nexport SOPS_AGE_RECIPIENTS=\\"age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p\\"\\n```plaintext ### Decryption Failed **Error**: ```plaintext\\nFailed to decrypt configuration file\\n```plaintext **Solutions**: 1. **Wrong key**: ```bash # Verify you have the correct private key provisioning config validate-encryption File corrupted : # Check file integrity\\nsops --decrypt workspace/config/secure.yaml Wrong backend : # Check SOPS metadata in file\\nhead -20 workspace/config/secure.yaml","breadcrumbs":"Config Encryption Guide » Verify Installation","id":"1526","title":"Verify Installation"},"1527":{"body":"Error : AccessDeniedException: User is not authorized to perform: kms:Decrypt\\n```plaintext **Solution**: ```bash\\n# Check AWS credentials\\naws sts get-caller-identity # Verify KMS key policy allows your IAM user/role\\naws kms describe-key --key-id \\n```plaintext ### Vault Connection Failed **Error**: ```plaintext\\nVault encryption failed: connection refused\\n```plaintext **Solution**: ```bash\\n# Verify Vault address\\necho $VAULT_ADDR # Check connectivity\\ncurl -k $VAULT_ADDR/v1/sys/health # Verify token\\nvault token lookup\\n```plaintext --- ## Security Considerations ### Threat Model **Protected Against**: - ✅ Plaintext secrets in git\\n- ✅ Accidental secret exposure\\n- ✅ Unauthorized file access\\n- ✅ Key compromise (with rotation) **Not Protected Against**: - ❌ Memory dumps during decryption\\n- ❌ Root/admin access to running process\\n- ❌ Compromised Age/KMS keys\\n- ❌ Social engineering ### Security Best Practices 1. **Principle of Least Privilege**: Only grant decryption access to those who need it\\n2. **Key Separation**: Use different keys for different environments\\n3. **Regular Audits**: Review who has access to keys\\n4. **Secure Key Storage**: Never store private keys in git\\n5. **Rotation**: Regularly rotate encryption keys\\n6. **Monitoring**: Monitor decryption operations (with AWS KMS/Vault) --- ## Additional Resources - **SOPS Documentation**: \\n- **Age Encryption**: \\n- **AWS KMS**: \\n- **HashiCorp Vault**: \\n- **Cosmian KMS**:  --- ## Support For issues or questions: - Check troubleshooting section above\\n- Run: `provisioning config validate-encryption`\\n- Review logs with `--debug` flag --- ## Quick Reference ### Setup (One-time) ```bash\\n# 1. Initialize encryption\\nprovisioning config init-encryption --kms age # 2. Set environment variables (add to ~/.zshrc or ~/.bashrc)\\nexport SOPS_AGE_RECIPIENTS=\\"age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p\\"\\nexport PROVISIONING_KAGE=\\"$HOME/.config/sops/age/keys.txt\\" # 3. Validate setup\\nprovisioning config validate-encryption\\n```plaintext ### Common Commands | Task | Command |\\n|------|---------|\\n| **Encrypt file** | `provisioning config encrypt secrets.yaml --in-place` |\\n| **Decrypt file** | `provisioning config decrypt secrets.enc.yaml` |\\n| **Edit encrypted** | `provisioning config edit-secure secrets.enc.yaml` |\\n| **Check if encrypted** | `provisioning config is-encrypted secrets.yaml` |\\n| **Scan for unencrypted** | `provisioning config scan-sensitive workspace --recursive` |\\n| **Encrypt all sensitive** | `provisioning config encrypt-all workspace/config --kms age` |\\n| **Validate setup** | `provisioning config validate-encryption` |\\n| **Show encryption info** | `provisioning config encryption-info secrets.yaml` | ### File Naming Conventions Automatically encrypted by SOPS: - `workspace/*/config/secure.yaml` ← Auto-encrypted\\n- `*.enc.yaml` ← Auto-encrypted\\n- `*.enc.yml` ← Auto-encrypted\\n- `*.enc.toml` ← Auto-encrypted\\n- `workspace/*/config/providers/*credentials*.toml` ← Auto-encrypted ### Quick Workflow ```bash\\n# Create config with secrets\\ncat > workspace/config/secure.yaml < edit -> re-encrypt)\\nprovisioning config edit-secure workspace/config/secure.yaml # Configs are auto-decrypted when loaded\\nprovisioning env # Automatically decrypts secure.yaml\\n```plaintext ### KMS Backends | Backend | Use Case | Setup Command |\\n|---------|----------|---------------|\\n| **Age** | Development, simple setup | `provisioning config init-encryption --kms age` |\\n| **AWS KMS** | Production, AWS environments | Configure in `.sops.yaml` |\\n| **Vault** | Enterprise, dynamic secrets | Set `VAULT_ADDR` and `VAULT_TOKEN` |\\n| **Cosmian** | Confidential computing | Configure in `config.toml` | ### Security Checklist - ✅ Encrypt all files with passwords, API keys, secrets\\n- ✅ Never commit unencrypted secrets to git\\n- ✅ Set file permissions: `chmod 600 ~/.config/sops/age/keys.txt`\\n- ✅ Add plaintext files to `.gitignore`: `*.dec.yaml`, `secrets.yaml`\\n- ✅ Regular key rotation (quarterly for production)\\n- ✅ Separate keys per environment (dev/staging/prod)\\n- ✅ Backup Age keys securely (encrypted backup) ### Troubleshooting | Problem | Solution |\\n|---------|----------|\\n| `SOPS binary not found` | `brew install sops` |\\n| `Age key file not found` | `provisioning config init-encryption --kms age` |\\n| `SOPS_AGE_RECIPIENTS not set` | `export SOPS_AGE_RECIPIENTS=\\"age1...\\"` |\\n| `Decryption failed` | Check key file: `provisioning config validate-encryption` |\\n| `AWS KMS Access Denied` | Verify IAM permissions: `aws sts get-caller-identity` | ### Testing ```bash\\n# Run all encryption tests\\nnu provisioning/core/nulib/lib_provisioning/config/encryption_tests.nu # Run specific test\\nnu provisioning/core/nulib/lib_provisioning/config/encryption_tests.nu --test roundtrip # Test full workflow\\nnu provisioning/core/nulib/lib_provisioning/config/encryption_tests.nu test-full-encryption-workflow # Test KMS backend\\nuse lib_provisioning/kms/client.nu\\nkms-test --backend age\\n```plaintext ### Integration Configs are **automatically decrypted** when loaded: ```nushell\\n# Nushell code - encryption is transparent\\nuse lib_provisioning/config/loader.nu # Auto-decrypts encrypted files in memory\\nlet config = (load-provisioning-config) # Access secrets normally\\nlet db_password = ($config | get database.password)\\n```plaintext ### Emergency Key Recovery If you lose your Age key: 1. **Check backups**: `~/.config/sops/age/keys.txt.backup`\\n2. **Check other systems**: Keys might be on other dev machines\\n3. **Contact team**: Team members with access can re-encrypt for you\\n4. **Rotate secrets**: If keys are lost, rotate all secrets ### Advanced #### Multiple Recipients (Team Access) ```yaml\\n# .sops.yaml\\ncreation_rules: - path_regex: .*\\\\.enc\\\\.yaml$ age: >- age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8q\\n```plaintext #### Key Rotation ```bash\\n# Generate new key\\nage-keygen -o ~/.config/sops/age/keys-new.txt # Update .sops.yaml with new recipient # Rotate keys for file\\nprovisioning config rotate-keys workspace/config/secure.yaml \\n```plaintext #### Scan and Encrypt All ```bash\\n# Find all unencrypted sensitive configs\\nprovisioning config scan-sensitive workspace --recursive # Encrypt them all\\nprovisioning config encrypt-all workspace --kms age --recursive # Verify\\nprovisioning config scan-sensitive workspace --recursive\\n```plaintext ### Documentation - **Full Guide**: `docs/user/CONFIG_ENCRYPTION_GUIDE.md`\\n- **SOPS Docs**: \\n- **Age Docs**:  --- **Last Updated**: 2025-10-08\\n**Version**: 1.0.0","breadcrumbs":"Config Encryption Guide » AWS KMS Access Denied","id":"1527","title":"AWS KMS Access Denied"},"1528":{"body":"","breadcrumbs":"Security System » Complete Security System (v4.0.0)","id":"1528","title":"Complete Security System (v4.0.0)"},"1529":{"body":"A comprehensive security system with 39,699 lines across 12 components providing enterprise-grade protection for infrastructure automation.","breadcrumbs":"Security System » 🔐 Enterprise-Grade Security Implementation","id":"1529","title":"🔐 Enterprise-Grade Security Implementation"},"153":{"body":"","breadcrumbs":"Installation Steps » Troubleshooting","id":"153","title":"Troubleshooting"},"1530":{"body":"","breadcrumbs":"Security System » Core Security Components","id":"1530","title":"Core Security Components"},"1531":{"body":"Type : RS256 token-based authentication Features : Argon2id hashing, token rotation, session management Roles : 5 distinct role levels with inheritance Commands : provisioning login\\nprovisioning mfa totp verify","breadcrumbs":"Security System » 1. Authentication (JWT)","id":"1531","title":"1. Authentication (JWT)"},"1532":{"body":"Type : Policy-as-code using Cedar authorization engine Features : Context-aware policies, hot reload, fine-grained control Updates : Dynamic policy reloading without service restart","breadcrumbs":"Security System » 2. Authorization (Cedar)","id":"1532","title":"2. Authorization (Cedar)"},"1533":{"body":"Methods : TOTP (Time-based OTP) + WebAuthn/FIDO2 Features : Backup codes, rate limiting, device binding Commands : provisioning mfa totp enroll\\nprovisioning mfa webauthn enroll","breadcrumbs":"Security System » 3. Multi-Factor Authentication (MFA)","id":"1533","title":"3. Multi-Factor Authentication (MFA)"},"1534":{"body":"Dynamic Secrets : AWS STS, SSH keys, UpCloud credentials KMS Integration : Vault + AWS KMS + Age + Cosmian Features : Auto-cleanup, TTL management, rotation policies Commands : provisioning secrets generate aws --ttl 1hr\\nprovisioning ssh connect server01","breadcrumbs":"Security System » 4. Secrets Management","id":"1534","title":"4. Secrets Management"},"1535":{"body":"Backends : RustyVault, Age, AWS KMS, HashiCorp Vault, Cosmian Features : Envelope encryption, key rotation, secure storage Commands : provisioning kms encrypt\\nprovisioning config encrypt secure.yaml","breadcrumbs":"Security System » 5. Key Management System (KMS)","id":"1535","title":"5. Key Management System (KMS)"},"1536":{"body":"Format : Structured JSON logs with full context Compliance : GDPR-compliant with PII filtering Retention : 7-year data retention policy Exports : 5 export formats (JSON, CSV, SYSLOG, Splunk, CloudWatch)","breadcrumbs":"Security System » 6. Audit Logging","id":"1536","title":"6. Audit Logging"},"1537":{"body":"Approval : Multi-party approval workflow Features : Temporary elevated privileges, auto-revocation, audit trail Commands : provisioning break-glass request \\"reason\\"\\nprovisioning break-glass approve ","breadcrumbs":"Security System » 7. Break-Glass Emergency Access","id":"1537","title":"7. Break-Glass Emergency Access"},"1538":{"body":"Standards : GDPR, SOC2, ISO 27001, incident response procedures Features : Compliance reporting, audit trails, policy enforcement Commands : provisioning compliance report\\nprovisioning compliance gdpr export ","breadcrumbs":"Security System » 8. Compliance Management","id":"1538","title":"8. Compliance Management"},"1539":{"body":"Filtering : By user, action, time range, resource Features : Structured query language, real-time search Commands : provisioning audit query --user alice --action deploy --from 24h","breadcrumbs":"Security System » 9. Audit Query System","id":"1539","title":"9. Audit Query System"},"154":{"body":"If plugins aren\'t recognized: # Rebuild plugin registry\\nnu -c \\"plugin list; plugin use tera\\"","breadcrumbs":"Installation Steps » Nushell Plugin Not Found","id":"154","title":"Nushell Plugin Not Found"},"1540":{"body":"Features : Rotation policies, expiration tracking, revocation Integration : Seamless with auth system","breadcrumbs":"Security System » 10. Token Management","id":"1540","title":"10. Token Management"},"1541":{"body":"Model : Role-based access control (RBAC) Features : Resource-level permissions, delegation, audit","breadcrumbs":"Security System » 11. Access Control","id":"1541","title":"11. Access Control"},"1542":{"body":"Standards : AES-256, TLS 1.3, envelope encryption Coverage : At-rest and in-transit encryption","breadcrumbs":"Security System » 12. Encryption","id":"1542","title":"12. Encryption"},"1543":{"body":"Overhead : <20ms per secure operation Tests : 350+ comprehensive test cases Endpoints : 83+ REST API endpoints CLI Commands : 111+ security-related commands","breadcrumbs":"Security System » Performance Characteristics","id":"1543","title":"Performance Characteristics"},"1544":{"body":"Component Command Purpose Login provisioning login User authentication MFA TOTP provisioning mfa totp enroll Setup time-based MFA MFA WebAuthn provisioning mfa webauthn enroll Setup hardware security key Secrets provisioning secrets generate aws --ttl 1hr Generate temporary credentials SSH provisioning ssh connect server01 Secure SSH session KMS Encrypt provisioning kms encrypt  Encrypt configuration Break-Glass provisioning break-glass request \\"reason\\" Request emergency access Compliance provisioning compliance report Generate compliance report GDPR Export provisioning compliance gdpr export  Export user data Audit provisioning audit query --user alice --action deploy --from 24h Search audit logs","breadcrumbs":"Security System » Quick Reference","id":"1544","title":"Quick Reference"},"1545":{"body":"Security system is integrated throughout provisioning platform: Embedded : All authentication/authorization checks Non-blocking : <20ms overhead on operations Graceful degradation : Fallback mechanisms for partial failures Hot reload : Policies update without service restart","breadcrumbs":"Security System » Architecture","id":"1545","title":"Architecture"},"1546":{"body":"Security policies and settings are defined in: provisioning/kcl/security.k - KCL security schema definitions provisioning/config/security/*.toml - Security policy configurations Environment-specific overrides in workspace/config/","breadcrumbs":"Security System » Configuration","id":"1546","title":"Configuration"},"1547":{"body":"Full implementation: ADR-009: Security System Complete User guides: Authentication Layer Guide Admin guides: MFA Admin Setup Guide Implementation details: Various documentation subdirectories","breadcrumbs":"Security System » Documentation","id":"1547","title":"Documentation"},"1548":{"body":"# Show security help\\nprovisioning help security # Show specific security command help\\nprovisioning login --help\\nprovisioning mfa --help\\nprovisioning secrets --help","breadcrumbs":"Security System » Help Commands","id":"1548","title":"Help Commands"},"1549":{"body":"Version : 1.0.0 Date : 2025-10-08 Status : Production-ready","breadcrumbs":"RustyVault KMS Guide » RustyVault KMS Backend Guide","id":"1549","title":"RustyVault KMS Backend Guide"},"155":{"body":"If you encounter permission errors: # Ensure proper ownership\\nsudo chown -R $USER:$USER ~/.config/provisioning # Check PATH\\necho $PATH | grep provisioning","breadcrumbs":"Installation Steps » Permission Denied","id":"155","title":"Permission Denied"},"1550":{"body":"RustyVault is a self-hosted, Rust-based secrets management system that provides a Vault-compatible API . The provisioning platform now supports RustyVault as a KMS backend alongside Age, Cosmian, AWS KMS, and HashiCorp Vault.","breadcrumbs":"RustyVault KMS Guide » Overview","id":"1550","title":"Overview"},"1551":{"body":"Self-hosted : Full control over your key management infrastructure Pure Rust : Better performance and memory safety Vault-compatible : Drop-in replacement for HashiCorp Vault Transit engine OSI-approved License : Apache 2.0 (vs HashiCorp\'s BSL) Embeddable : Can run as standalone service or embedded library No Vendor Lock-in : Open-source alternative to proprietary KMS solutions","breadcrumbs":"RustyVault KMS Guide » Why RustyVault?","id":"1551","title":"Why RustyVault?"},"1552":{"body":"KMS Service Backends:\\n├── Age (local development, file-based)\\n├── Cosmian (privacy-preserving, production)\\n├── AWS KMS (cloud-native AWS)\\n├── HashiCorp Vault (enterprise, external)\\n└── RustyVault (self-hosted, embedded) ✨ NEW\\n```plaintext --- ## Installation ### Option 1: Standalone RustyVault Server ```bash\\n# Install RustyVault binary\\ncargo install rusty_vault # Start RustyVault server\\nrustyvault server -config=/path/to/config.hcl\\n```plaintext ### Option 2: Docker Deployment ```bash\\n# Pull RustyVault image (if available)\\ndocker pull tongsuo/rustyvault:latest # Run RustyVault container\\ndocker run -d \\\\ --name rustyvault \\\\ -p 8200:8200 \\\\ -v $(pwd)/config:/vault/config \\\\ -v $(pwd)/data:/vault/data \\\\ tongsuo/rustyvault:latest\\n```plaintext ### Option 3: From Source ```bash\\n# Clone repository\\ngit clone https://github.com/Tongsuo-Project/RustyVault.git\\ncd RustyVault # Build and run\\ncargo build --release\\n./target/release/rustyvault server -config=config.hcl\\n```plaintext --- ## Configuration ### RustyVault Server Configuration Create `rustyvault-config.hcl`: ```hcl\\n# RustyVault Server Configuration storage \\"file\\" { path = \\"/vault/data\\"\\n} listener \\"tcp\\" { address = \\"0.0.0.0:8200\\" tls_disable = true # Enable TLS in production\\n} api_addr = \\"http://127.0.0.1:8200\\"\\ncluster_addr = \\"https://127.0.0.1:8201\\" # Enable Transit secrets engine\\ndefault_lease_ttl = \\"168h\\"\\nmax_lease_ttl = \\"720h\\"\\n```plaintext ### Initialize RustyVault ```bash\\n# Initialize (first time only)\\nexport VAULT_ADDR=\'http://127.0.0.1:8200\'\\nrustyvault operator init # Unseal (after every restart)\\nrustyvault operator unseal \\nrustyvault operator unseal \\nrustyvault operator unseal  # Save root token\\nexport RUSTYVAULT_TOKEN=\'\'\\n```plaintext ### Enable Transit Engine ```bash\\n# Enable transit secrets engine\\nrustyvault secrets enable transit # Create encryption key\\nrustyvault write -f transit/keys/provisioning-main # Verify key creation\\nrustyvault read transit/keys/provisioning-main\\n```plaintext --- ## KMS Service Configuration ### Update `provisioning/config/kms.toml` ```toml\\n[kms]\\ntype = \\"rustyvault\\"\\nserver_url = \\"http://localhost:8200\\"\\ntoken = \\"${RUSTYVAULT_TOKEN}\\"\\nmount_point = \\"transit\\"\\nkey_name = \\"provisioning-main\\"\\ntls_verify = true [service]\\nbind_addr = \\"0.0.0.0:8081\\"\\nlog_level = \\"info\\"\\naudit_logging = true [tls]\\nenabled = false # Set true with HTTPS\\n```plaintext ### Environment Variables ```bash\\n# RustyVault connection\\nexport RUSTYVAULT_ADDR=\\"http://localhost:8200\\"\\nexport RUSTYVAULT_TOKEN=\\"s.xxxxxxxxxxxxxxxxxxxxxx\\"\\nexport RUSTYVAULT_MOUNT_POINT=\\"transit\\"\\nexport RUSTYVAULT_KEY_NAME=\\"provisioning-main\\"\\nexport RUSTYVAULT_TLS_VERIFY=\\"true\\" # KMS service\\nexport KMS_BACKEND=\\"rustyvault\\"\\nexport KMS_BIND_ADDR=\\"0.0.0.0:8081\\"\\n```plaintext --- ## Usage ### Start KMS Service ```bash\\n# With RustyVault backend\\ncd provisioning/platform/kms-service\\ncargo run # With custom config\\ncargo run -- --config=/path/to/kms.toml\\n```plaintext ### CLI Operations ```bash\\n# Encrypt configuration file\\nprovisioning kms encrypt provisioning/config/secrets.yaml # Decrypt configuration\\nprovisioning kms decrypt provisioning/config/secrets.yaml.enc # Generate data key (envelope encryption)\\nprovisioning kms generate-key --spec AES256 # Health check\\nprovisioning kms health\\n```plaintext ### REST API Usage ```bash\\n# Health check\\ncurl http://localhost:8081/health # Encrypt data\\ncurl -X POST http://localhost:8081/encrypt \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{ \\"plaintext\\": \\"SGVsbG8sIFdvcmxkIQ==\\", \\"context\\": \\"environment=production\\" }\' # Decrypt data\\ncurl -X POST http://localhost:8081/decrypt \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{ \\"ciphertext\\": \\"vault:v1:...\\", \\"context\\": \\"environment=production\\" }\' # Generate data key\\ncurl -X POST http://localhost:8081/datakey/generate \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{\\"key_spec\\": \\"AES_256\\"}\'\\n```plaintext --- ## Advanced Features ### Context-based Encryption (AAD) Additional authenticated data binds encrypted data to specific contexts: ```bash\\n# Encrypt with context\\ncurl -X POST http://localhost:8081/encrypt \\\\ -d \'{ \\"plaintext\\": \\"c2VjcmV0\\", \\"context\\": \\"environment=prod,service=api\\" }\' # Decrypt requires same context\\ncurl -X POST http://localhost:8081/decrypt \\\\ -d \'{ \\"ciphertext\\": \\"vault:v1:...\\", \\"context\\": \\"environment=prod,service=api\\" }\'\\n```plaintext ### Envelope Encryption For large files, use envelope encryption: ```bash\\n# 1. Generate data key\\nDATA_KEY=$(curl -X POST http://localhost:8081/datakey/generate \\\\ -d \'{\\"key_spec\\": \\"AES_256\\"}\' | jq -r \'.plaintext\') # 2. Encrypt large file with data key (locally)\\nopenssl enc -aes-256-cbc -in large-file.bin -out encrypted.bin -K $DATA_KEY # 3. Store encrypted data key (from response)\\necho \\"vault:v1:...\\" > encrypted-data-key.txt\\n```plaintext ### Key Rotation ```bash\\n# Rotate encryption key in RustyVault\\nrustyvault write -f transit/keys/provisioning-main/rotate # Verify new version\\nrustyvault read transit/keys/provisioning-main # Rewrap existing ciphertext with new key version\\ncurl -X POST http://localhost:8081/rewrap \\\\ -d \'{\\"ciphertext\\": \\"vault:v1:...\\"}\'\\n```plaintext --- ## Production Deployment ### High Availability Setup Deploy multiple RustyVault instances behind a load balancer: ```yaml\\n# docker-compose.yml\\nversion: \'3.8\' services: rustyvault-1: image: tongsuo/rustyvault:latest ports: - \\"8200:8200\\" volumes: - ./config:/vault/config - vault-data-1:/vault/data rustyvault-2: image: tongsuo/rustyvault:latest ports: - \\"8201:8200\\" volumes: - ./config:/vault/config - vault-data-2:/vault/data lb: image: nginx:alpine ports: - \\"80:80\\" volumes: - ./nginx.conf:/etc/nginx/nginx.conf depends_on: - rustyvault-1 - rustyvault-2 volumes: vault-data-1: vault-data-2:\\n```plaintext ### TLS Configuration ```toml\\n# kms.toml\\n[kms]\\ntype = \\"rustyvault\\"\\nserver_url = \\"https://vault.example.com:8200\\"\\ntoken = \\"${RUSTYVAULT_TOKEN}\\"\\ntls_verify = true [tls]\\nenabled = true\\ncert_path = \\"/etc/kms/certs/server.crt\\"\\nkey_path = \\"/etc/kms/certs/server.key\\"\\nca_path = \\"/etc/kms/certs/ca.crt\\"\\n```plaintext ### Auto-Unseal (AWS KMS) ```hcl\\n# rustyvault-config.hcl\\nseal \\"awskms\\" { region = \\"us-east-1\\" kms_key_id = \\"arn:aws:kms:us-east-1:123456789012:key/...\\"\\n}\\n```plaintext --- ## Monitoring ### Health Checks ```bash\\n# RustyVault health\\ncurl http://localhost:8200/v1/sys/health # KMS service health\\ncurl http://localhost:8081/health # Metrics (if enabled)\\ncurl http://localhost:8081/metrics\\n```plaintext ### Audit Logging Enable audit logging in RustyVault: ```hcl\\n# rustyvault-config.hcl\\naudit { path = \\"/vault/logs/audit.log\\" format = \\"json\\"\\n}\\n```plaintext --- ## Troubleshooting ### Common Issues **1. Connection Refused** ```bash\\n# Check RustyVault is running\\ncurl http://localhost:8200/v1/sys/health # Check token is valid\\nexport VAULT_ADDR=\'http://localhost:8200\'\\nrustyvault token lookup\\n```plaintext **2. Authentication Failed** ```bash\\n# Verify token in environment\\necho $RUSTYVAULT_TOKEN # Renew token if needed\\nrustyvault token renew\\n```plaintext **3. Key Not Found** ```bash\\n# List available keys\\nrustyvault list transit/keys # Create missing key\\nrustyvault write -f transit/keys/provisioning-main\\n```plaintext **4. TLS Verification Failed** ```bash\\n# Disable TLS verification (dev only)\\nexport RUSTYVAULT_TLS_VERIFY=false # Or add CA certificate\\nexport RUSTYVAULT_CACERT=/path/to/ca.crt\\n```plaintext --- ## Migration from Other Backends ### From HashiCorp Vault RustyVault is API-compatible, minimal changes required: ```bash\\n# Old config (Vault)\\n[kms]\\ntype = \\"vault\\"\\naddress = \\"https://vault.example.com:8200\\"\\ntoken = \\"${VAULT_TOKEN}\\" # New config (RustyVault)\\n[kms]\\ntype = \\"rustyvault\\"\\nserver_url = \\"http://rustyvault.example.com:8200\\"\\ntoken = \\"${RUSTYVAULT_TOKEN}\\"\\n```plaintext ### From Age Re-encrypt existing encrypted files: ```bash\\n# 1. Decrypt with Age\\nprovisioning kms decrypt --backend age secrets.enc > secrets.plain # 2. Encrypt with RustyVault\\nprovisioning kms encrypt --backend rustyvault secrets.plain > secrets.rustyvault.enc\\n```plaintext --- ## Security Considerations ### Best Practices 1. **Enable TLS**: Always use HTTPS in production\\n2. **Rotate Tokens**: Regularly rotate RustyVault tokens\\n3. **Least Privilege**: Use policies to restrict token permissions\\n4. **Audit Logging**: Enable and monitor audit logs\\n5. **Backup Keys**: Secure backup of unseal keys and root token\\n6. **Network Isolation**: Run RustyVault in isolated network segment ### Token Policies Create restricted policy for KMS service: ```hcl\\n# kms-policy.hcl\\npath \\"transit/encrypt/provisioning-main\\" { capabilities = [\\"update\\"]\\n} path \\"transit/decrypt/provisioning-main\\" { capabilities = [\\"update\\"]\\n} path \\"transit/datakey/plaintext/provisioning-main\\" { capabilities = [\\"update\\"]\\n}\\n```plaintext Apply policy: ```bash\\nrustyvault policy write kms-service kms-policy.hcl\\nrustyvault token create -policy=kms-service\\n```plaintext --- ## Performance ### Benchmarks (Estimated) | Operation | Latency | Throughput |\\n|-----------|---------|------------|\\n| Encrypt | 5-15ms | 2,000-5,000 ops/sec |\\n| Decrypt | 5-15ms | 2,000-5,000 ops/sec |\\n| Generate Key | 10-20ms | 1,000-2,000 ops/sec | *Actual performance depends on hardware, network, and RustyVault configuration* ### Optimization Tips 1. **Connection Pooling**: Reuse HTTP connections\\n2. **Batching**: Batch multiple operations when possible\\n3. **Caching**: Cache data keys for envelope encryption\\n4. **Local Unseal**: Use auto-unseal for faster restarts --- ## Related Documentation - **KMS Service**: `docs/user/CONFIG_ENCRYPTION_GUIDE.md`\\n- **Dynamic Secrets**: `docs/user/DYNAMIC_SECRETS_QUICK_REFERENCE.md`\\n- **Security System**: `docs/architecture/ADR-009-security-system-complete.md`\\n- **RustyVault GitHub**:  --- ## Support - **GitHub Issues**: \\n- **Documentation**: \\n- **Community**:  --- **Last Updated**: 2025-10-08\\n**Maintained By**: Architecture Team","breadcrumbs":"RustyVault KMS Guide » Architecture Position","id":"1552","title":"Architecture Position"},"1553":{"body":"SecretumVault is an enterprise-grade, post-quantum ready secrets management system integrated as the 4th KMS backend in the provisioning platform, alongside Age (dev), Cosmian (prod), and RustyVault (self-hosted).","breadcrumbs":"SecretumVault KMS Guide » SecretumVault KMS Backend Guide","id":"1553","title":"SecretumVault KMS Backend Guide"},"1554":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Overview","id":"1554","title":"Overview"},"1555":{"body":"SecretumVault provides: Post-Quantum Cryptography : Ready for quantum-resistant algorithms Enterprise Features : Policy-as-code (Cedar), audit logging, compliance tracking Multiple Storage Backends : Filesystem (dev), SurrealDB (staging), etcd (prod), PostgreSQL Transit Engine : Encryption-as-a-service for data protection KV Engine : Versioned secret storage with rotation policies High Availability : Seamless transition from embedded to distributed modes","breadcrumbs":"SecretumVault KMS Guide » What is SecretumVault?","id":"1555","title":"What is SecretumVault?"},"1556":{"body":"Scenario Backend Reason Local development Age Simple, no dependencies Testing/Staging SecretumVault Enterprise features, production-like Production Cosmian or SecretumVault Enterprise security, compliance Self-Hosted Enterprise SecretumVault + etcd Full control, HA support","breadcrumbs":"SecretumVault KMS Guide » When to Use SecretumVault","id":"1556","title":"When to Use SecretumVault"},"1557":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Deployment Modes","id":"1557","title":"Deployment Modes"},"1558":{"body":"Storage : Filesystem (~/.config/provisioning/secretumvault/data) Performance : <3ms encryption/decryption Setup : No separate service required Best For : Local development and testing export PROVISIONING_ENV=dev\\nexport KMS_DEV_BACKEND=secretumvault\\nprovisioning kms encrypt config.yaml","breadcrumbs":"SecretumVault KMS Guide » Development Mode (Embedded)","id":"1558","title":"Development Mode (Embedded)"},"1559":{"body":"Storage : SurrealDB (document database) Performance : <10ms operations Setup : Start SecretumVault service separately Best For : Team testing, staging environments # Start SecretumVault service\\nsecretumvault server --storage-backend surrealdb # Configure provisioning\\nexport PROVISIONING_ENV=staging\\nexport SECRETUMVAULT_URL=http://localhost:8200\\nexport SECRETUMVAULT_TOKEN=your-auth-token provisioning kms encrypt config.yaml","breadcrumbs":"SecretumVault KMS Guide » Staging Mode (Service + SurrealDB)","id":"1559","title":"Staging Mode (Service + SurrealDB)"},"156":{"body":"If encryption fails: # Verify keys exist\\nls -la ~/.config/provisioning/age/ # Regenerate if needed\\nage-keygen -o ~/.config/provisioning/age/private_key.txt","breadcrumbs":"Installation Steps » Age Keys Not Found","id":"156","title":"Age Keys Not Found"},"1560":{"body":"Storage : etcd cluster (3+ nodes) Performance : <10ms operations (99th percentile) Setup : etcd cluster + SecretumVault service Best For : Production deployments with HA requirements # Setup etcd cluster (3 nodes minimum)\\netcd --name etcd1 --data-dir etcd1-data \\\\ --advertise-client-urls http://localhost:2379 \\\\ --listen-client-urls http://localhost:2379 # Start SecretumVault with etcd\\nsecretumvault server \\\\ --storage-backend etcd \\\\ --etcd-endpoints http://etcd1:2379,http://etcd2:2379,http://etcd3:2379 # Configure provisioning\\nexport PROVISIONING_ENV=prod\\nexport SECRETUMVAULT_URL=https://your-secretumvault:8200\\nexport SECRETUMVAULT_TOKEN=your-auth-token\\nexport SECRETUMVAULT_STORAGE=etcd provisioning kms encrypt config.yaml","breadcrumbs":"SecretumVault KMS Guide » Production Mode (Service + etcd)","id":"1560","title":"Production Mode (Service + etcd)"},"1561":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Configuration","id":"1561","title":"Configuration"},"1562":{"body":"Variable Purpose Default Example PROVISIONING_ENV Deployment environment dev staging, prod KMS_DEV_BACKEND Development KMS backend age secretumvault KMS_STAGING_BACKEND Staging KMS backend secretumvault cosmian KMS_PROD_BACKEND Production KMS backend cosmian secretumvault SECRETUMVAULT_URL Server URL http://localhost:8200 https://kms.example.com SECRETUMVAULT_TOKEN Authentication token (none) (Bearer token) SECRETUMVAULT_STORAGE Storage backend filesystem surrealdb, etcd SECRETUMVAULT_TLS_VERIFY Verify TLS certificates false true","breadcrumbs":"SecretumVault KMS Guide » Environment Variables","id":"1562","title":"Environment Variables"},"1563":{"body":"System Defaults : provisioning/config/secretumvault.toml KMS Config : provisioning/config/kms.toml Edit these files to customize: Engine mount points Key names Storage backend settings Performance tuning Audit logging Key rotation policies","breadcrumbs":"SecretumVault KMS Guide » Configuration Files","id":"1563","title":"Configuration Files"},"1564":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Operations","id":"1564","title":"Operations"},"1565":{"body":"# Encrypt a file\\nprovisioning kms encrypt config.yaml\\n# Output: config.yaml.enc # Encrypt with specific key\\nprovisioning kms encrypt --key-id my-key config.yaml # Encrypt and sign\\nprovisioning kms encrypt --sign config.yaml","breadcrumbs":"SecretumVault KMS Guide » Encrypt Data","id":"1565","title":"Encrypt Data"},"1566":{"body":"# Decrypt a file\\nprovisioning kms decrypt config.yaml.enc\\n# Output: config.yaml # Decrypt with specific key\\nprovisioning kms decrypt --key-id my-key config.yaml.enc # Verify and decrypt\\nprovisioning kms decrypt --verify config.yaml.enc","breadcrumbs":"SecretumVault KMS Guide » Decrypt Data","id":"1566","title":"Decrypt Data"},"1567":{"body":"# Generate AES-256 data key\\nprovisioning kms generate-key --spec AES256 # Generate AES-128 data key\\nprovisioning kms generate-key --spec AES128 # Generate RSA-4096 key\\nprovisioning kms generate-key --spec RSA4096","breadcrumbs":"SecretumVault KMS Guide » Generate Data Keys","id":"1567","title":"Generate Data Keys"},"1568":{"body":"# Check KMS health\\nprovisioning kms health # Get KMS version\\nprovisioning kms version # Detailed KMS status\\nprovisioning kms status","breadcrumbs":"SecretumVault KMS Guide » Health and Status","id":"1568","title":"Health and Status"},"1569":{"body":"# Rotate encryption key\\nprovisioning kms rotate-key provisioning-master # Check rotation policy\\nprovisioning kms rotation-policy provisioning-master # Update rotation interval\\nprovisioning kms update-rotation 90 # Rotate every 90 days","breadcrumbs":"SecretumVault KMS Guide » Key Rotation","id":"1569","title":"Key Rotation"},"157":{"body":"Once installation is complete, proceed to: → First Deployment","breadcrumbs":"Installation Steps » Next Steps","id":"157","title":"Next Steps"},"1570":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Storage Backends","id":"1570","title":"Storage Backends"},"1571":{"body":"Local file-based storage with no external dependencies. Pros : Zero external dependencies Fast (local disk access) Easy to inspect/backup Cons : Single-node only No HA Manual backup required Configuration : [secretumvault.storage.filesystem]\\ndata_dir = \\"~/.config/provisioning/secretumvault/data\\"\\npermissions = \\"0700\\"","breadcrumbs":"SecretumVault KMS Guide » Filesystem (Development)","id":"1571","title":"Filesystem (Development)"},"1572":{"body":"Embedded or standalone document database. Pros : Embedded or distributed Flexible schema Real-time syncing Cons : More complex than filesystem New technology (less tested than etcd) Configuration : [secretumvault.storage.surrealdb]\\nconnection_url = \\"ws://localhost:8000\\"\\nnamespace = \\"provisioning\\"\\ndatabase = \\"secrets\\"\\nusername = \\"${SECRETUMVAULT_SURREALDB_USER:-admin}\\"\\npassword = \\"${SECRETUMVAULT_SURREALDB_PASS:-password}\\"","breadcrumbs":"SecretumVault KMS Guide » SurrealDB (Staging)","id":"1572","title":"SurrealDB (Staging)"},"1573":{"body":"Distributed key-value store for high availability. Pros : Proven in production HA and disaster recovery Consistent consensus protocol Multi-site replication Cons : Operational complexity Requires 3+ nodes More infrastructure Configuration : [secretumvault.storage.etcd]\\nendpoints = [\\"http://etcd1:2379\\", \\"http://etcd2:2379\\", \\"http://etcd3:2379\\"]\\ntls_enabled = true\\ntls_cert_file = \\"/path/to/client.crt\\"\\ntls_key_file = \\"/path/to/client.key\\"","breadcrumbs":"SecretumVault KMS Guide » etcd (Production)","id":"1573","title":"etcd (Production)"},"1574":{"body":"Relational database backend. Pros : Mature and reliable Advanced querying Full ACID transactions Cons : Schema requirements External database dependency More operational overhead Configuration : [secretumvault.storage.postgresql]\\nconnection_url = \\"postgresql://user:pass@localhost:5432/secretumvault\\"\\nmax_connections = 10\\nssl_mode = \\"require\\"","breadcrumbs":"SecretumVault KMS Guide » PostgreSQL (Enterprise)","id":"1574","title":"PostgreSQL (Enterprise)"},"1575":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Troubleshooting","id":"1575","title":"Troubleshooting"},"1576":{"body":"Error : \\"Failed to connect to SecretumVault service\\" Solutions : Verify SecretumVault is running: curl http://localhost:8200/v1/sys/health Check server URL configuration: provisioning config show secretumvault.server_url Verify network connectivity: nc -zv localhost 8200","breadcrumbs":"SecretumVault KMS Guide » Connection Errors","id":"1576","title":"Connection Errors"},"1577":{"body":"Error : \\"Authentication failed: X-Vault-Token missing or invalid\\" Solutions : Set authentication token: export SECRETUMVAULT_TOKEN=your-token Verify token is still valid: provisioning secrets verify-token Get new token from SecretumVault: secretumvault auth login","breadcrumbs":"SecretumVault KMS Guide » Authentication Failures","id":"1577","title":"Authentication Failures"},"1578":{"body":"Filesystem Backend Error : \\"Permission denied: ~/.config/provisioning/secretumvault/data\\" Solution : Check directory permissions: ls -la ~/.config/provisioning/secretumvault/\\n# Should be: drwx------ (0700)\\nchmod 700 ~/.config/provisioning/secretumvault/data SurrealDB Backend Error : \\"Failed to connect to SurrealDB at ws://localhost:8000\\" Solution : Start SurrealDB first: surreal start --bind 0.0.0.0:8000 file://secretum.db etcd Backend Error : \\"etcd cluster unhealthy\\" Solution : Check etcd cluster status: etcdctl member list\\netcdctl endpoint health # Verify all nodes are reachable\\ncurl http://etcd1:2379/health\\ncurl http://etcd2:2379/health\\ncurl http://etcd3:2379/health","breadcrumbs":"SecretumVault KMS Guide » Storage Backend Errors","id":"1578","title":"Storage Backend Errors"},"1579":{"body":"Slow encryption/decryption : Check network latency (for service mode): ping -c 3 secretumvault-server Monitor SecretumVault performance: provisioning kms metrics Check storage backend performance: Filesystem: Check disk I/O SurrealDB: Monitor database load etcd: Check cluster consensus state High memory usage : Check cache settings: provisioning config show secretumvault.performance.cache_ttl Reduce cache TTL: provisioning config set secretumvault.performance.cache_ttl 60 Monitor active connections: provisioning kms status","breadcrumbs":"SecretumVault KMS Guide » Performance Issues","id":"1579","title":"Performance Issues"},"158":{"body":"Detailed Installation Guide Workspace Management Troubleshooting Guide","breadcrumbs":"Installation Steps » Additional Resources","id":"158","title":"Additional Resources"},"1580":{"body":"Enable debug logging : export RUST_LOG=debug\\nprovisioning kms encrypt config.yaml Check configuration : provisioning config show secretumvault\\nprovisioning config validate Test connectivity : provisioning kms health --verbose View audit logs : tail -f ~/.config/provisioning/logs/secretumvault-audit.log","breadcrumbs":"SecretumVault KMS Guide » Debugging","id":"1580","title":"Debugging"},"1581":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Security Best Practices","id":"1581","title":"Security Best Practices"},"1582":{"body":"Never commit tokens to version control Use environment variables or .env files (gitignored) Rotate tokens regularly Use different tokens per environment","breadcrumbs":"SecretumVault KMS Guide » Token Management","id":"1582","title":"Token Management"},"1583":{"body":"Enable TLS verification in production: export SECRETUMVAULT_TLS_VERIFY=true Use proper certificates (not self-signed in production) Pin certificates to prevent MITM attacks","breadcrumbs":"SecretumVault KMS Guide » TLS/SSL","id":"1583","title":"TLS/SSL"},"1584":{"body":"Restrict who can access SecretumVault admin UI Use strong authentication (MFA preferred) Audit all secrets access Implement least-privilege principle","breadcrumbs":"SecretumVault KMS Guide » Access Control","id":"1584","title":"Access Control"},"1585":{"body":"Rotate keys regularly (every 90 days recommended) Keep old versions for decryption Test rotation procedures in staging first Monitor rotation status","breadcrumbs":"SecretumVault KMS Guide » Key Rotation","id":"1585","title":"Key Rotation"},"1586":{"body":"Backup SecretumVault data regularly Test restore procedures Store backups securely Keep backup keys separate from encrypted data","breadcrumbs":"SecretumVault KMS Guide » Backup and Recovery","id":"1586","title":"Backup and Recovery"},"1587":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Migration Guide","id":"1587","title":"Migration Guide"},"1588":{"body":"# Export all secrets encrypted with Age\\nprovisioning secrets export --backend age --output secrets.json # Import into SecretumVault\\nprovisioning secrets import --backend secretumvault secrets.json # Re-encrypt all configurations\\nfind workspace/infra -name \\"*.enc\\" -exec provisioning kms reencrypt {} \\\\;","breadcrumbs":"SecretumVault KMS Guide » From Age to SecretumVault","id":"1588","title":"From Age to SecretumVault"},"1589":{"body":"# Both use Vault-compatible APIs, so migration is simpler:\\n# 1. Ensure SecretumVault keys are available\\n# 2. Update KMS_PROD_BACKEND=secretumvault\\n# 3. Test with staging first\\n# 4. Monitor during transition","breadcrumbs":"SecretumVault KMS Guide » From RustyVault to SecretumVault","id":"1589","title":"From RustyVault to SecretumVault"},"159":{"body":"This guide walks you through deploying your first infrastructure using the Provisioning Platform.","breadcrumbs":"First Deployment » First Deployment","id":"159","title":"First Deployment"},"1590":{"body":"# For production migration:\\n# 1. Set up SecretumVault with etcd backend\\n# 2. Verify high availability is working\\n# 3. Run parallel encryption with both systems\\n# 4. Validate all decryptions work\\n# 5. Update KMS_PROD_BACKEND=secretumvault\\n# 6. Monitor closely for 24 hours\\n# 7. Keep Cosmian as fallback for 7 days","breadcrumbs":"SecretumVault KMS Guide » From Cosmian to SecretumVault","id":"1590","title":"From Cosmian to SecretumVault"},"1591":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Performance Tuning","id":"1591","title":"Performance Tuning"},"1592":{"body":"[secretumvault.performance]\\nmax_connections = 5\\nconnection_timeout = 5\\nrequest_timeout = 30\\ncache_ttl = 60","breadcrumbs":"SecretumVault KMS Guide » Development (Filesystem)","id":"1592","title":"Development (Filesystem)"},"1593":{"body":"[secretumvault.performance]\\nmax_connections = 20\\nconnection_timeout = 5\\nrequest_timeout = 30\\ncache_ttl = 300","breadcrumbs":"SecretumVault KMS Guide » Staging (SurrealDB)","id":"1593","title":"Staging (SurrealDB)"},"1594":{"body":"[secretumvault.performance]\\nmax_connections = 50\\nconnection_timeout = 10\\nrequest_timeout = 30\\ncache_ttl = 600","breadcrumbs":"SecretumVault KMS Guide » Production (etcd)","id":"1594","title":"Production (etcd)"},"1595":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Compliance and Audit","id":"1595","title":"Compliance and Audit"},"1596":{"body":"All operations are logged: # View recent audit events\\nprovisioning kms audit --limit 100 # Export audit logs\\nprovisioning kms audit export --output audit.json # Audit specific operations\\nprovisioning kms audit --action encrypt --from 24h","breadcrumbs":"SecretumVault KMS Guide » Audit Logging","id":"1596","title":"Audit Logging"},"1597":{"body":"# Generate compliance report\\nprovisioning compliance report --backend secretumvault # GDPR data export\\nprovisioning compliance gdpr-export user@example.com # SOC2 audit trail\\nprovisioning compliance soc2-export --output soc2-audit.json","breadcrumbs":"SecretumVault KMS Guide » Compliance Reports","id":"1597","title":"Compliance Reports"},"1598":{"body":"","breadcrumbs":"SecretumVault KMS Guide » Advanced Topics","id":"1598","title":"Advanced Topics"},"1599":{"body":"Enable fine-grained access control: # Enable Cedar integration\\nprovisioning config set secretumvault.authorization.cedar_enabled true # Define access policies\\nprovisioning policy define-kms-access user@example.com admin\\nprovisioning policy define-kms-access deployer@example.com deploy-only","breadcrumbs":"SecretumVault KMS Guide » Cedar Authorization Policies","id":"1599","title":"Cedar Authorization Policies"},"16":{"body":"Linux : Any modern distribution (Ubuntu 20.04+, CentOS 8+, Debian 11+) macOS : 11.0+ (Big Sur and newer) Windows : Windows 10/11 with WSL2","breadcrumbs":"Installation Guide » Operating System Support","id":"16","title":"Operating System Support"},"160":{"body":"In this chapter, you\'ll: Configure a simple infrastructure Create your first server Install a task service (Kubernetes) Verify the deployment Estimated time: 10-15 minutes","breadcrumbs":"First Deployment » Overview","id":"160","title":"Overview"},"1600":{"body":"Configure master key settings: # Set KEK rotation interval\\nprovisioning config set secretumvault.rotation.rotation_interval_days 90 # Enable automatic rotation\\nprovisioning config set secretumvault.rotation.auto_rotate true # Retain old versions for decryption\\nprovisioning config set secretumvault.rotation.retain_old_versions true","breadcrumbs":"SecretumVault KMS Guide » Key Encryption Keys (KEK)","id":"1600","title":"Key Encryption Keys (KEK)"},"1601":{"body":"For production deployments across regions: # Region 1\\nexport SECRETUMVAULT_URL=https://kms-us-east.example.com\\nexport SECRETUMVAULT_STORAGE=etcd # Region 2 (for failover)\\nexport SECRETUMVAULT_URL_FALLBACK=https://kms-us-west.example.com","breadcrumbs":"SecretumVault KMS Guide » Multi-Region Setup","id":"1601","title":"Multi-Region Setup"},"1602":{"body":"Documentation : docs/user/SECRETUMVAULT_KMS_GUIDE.md (this file) Configuration Template : provisioning/config/secretumvault.toml KMS Configuration : provisioning/config/kms.toml Issues : Report issues with provisioning kms debug Logs : Check ~/.config/provisioning/logs/secretumvault-*.log","breadcrumbs":"SecretumVault KMS Guide » Support and Resources","id":"1602","title":"Support and Resources"},"1603":{"body":"Age KMS Guide - Simple local encryption Cosmian KMS Guide - Enterprise confidential computing RustyVault Guide - Self-hosted Vault KMS Overview - KMS backend comparison","breadcrumbs":"SecretumVault KMS Guide » See Also","id":"1603","title":"See Also"},"1604":{"body":"","breadcrumbs":"SSH Temporal Keys User Guide » SSH Temporal Keys - User Guide","id":"1604","title":"SSH Temporal Keys - User Guide"},"1605":{"body":"","breadcrumbs":"SSH Temporal Keys User Guide » Quick Start","id":"1605","title":"Quick Start"},"1606":{"body":"The fastest way to use temporal SSH keys: # Auto-generate, deploy, and connect (key auto-revoked after disconnect)\\nssh connect server.example.com # Connect with custom user and TTL\\nssh connect server.example.com --user deploy --ttl 30min # Keep key active after disconnect\\nssh connect server.example.com --keep\\n```plaintext ### Manual Key Management For more control over the key lifecycle: ```bash\\n# 1. Generate key\\nssh generate-key server.example.com --user root --ttl 1hr # Output:\\n# ✓ SSH key generated successfully\\n# Key ID: abc-123-def-456\\n# Type: dynamickeypair\\n# User: root\\n# Server: server.example.com\\n# Expires: 2024-01-01T13:00:00Z\\n# Fingerprint: SHA256:...\\n#\\n# Private Key (save securely):\\n# -----BEGIN OPENSSH PRIVATE KEY-----\\n# ...\\n# -----END OPENSSH PRIVATE KEY----- # 2. Deploy key to server\\nssh deploy-key abc-123-def-456 # 3. Use the private key to connect\\nssh -i /path/to/private/key root@server.example.com # 4. Revoke when done\\nssh revoke-key abc-123-def-456\\n```plaintext ## Key Features ### Automatic Expiration All keys expire automatically after their TTL: - **Default TTL**: 1 hour\\n- **Configurable**: From 5 minutes to 24 hours\\n- **Background Cleanup**: Automatic removal from servers every 5 minutes ### Multiple Key Types Choose the right key type for your use case: | Type | Description | Use Case |\\n|------|-------------|----------|\\n| **dynamic** (default) | Generated Ed25519 keys | Quick SSH access |\\n| **ca** | Vault CA-signed certificate | Enterprise with SSH CA |\\n| **otp** | Vault one-time password | Single-use access | ### Security Benefits ✅ No static SSH keys to manage\\n✅ Short-lived credentials (1 hour default)\\n✅ Automatic cleanup on expiration\\n✅ Audit trail for all operations\\n✅ Private keys never stored on disk ## Common Usage Patterns ### Development Workflow ```bash\\n# Quick SSH for debugging\\nssh connect dev-server.local --ttl 30min # Execute commands\\nssh root@dev-server.local \\"systemctl status nginx\\" # Connection closes, key auto-revokes\\n```plaintext ### Production Deployment ```bash\\n# Generate key with longer TTL for deployment\\nssh generate-key prod-server.example.com --ttl 2hr # Deploy to server\\nssh deploy-key  # Run deployment script\\nssh -i /tmp/deploy-key root@prod-server.example.com < deploy.sh # Manual revoke when done\\nssh revoke-key \\n```plaintext ### Multi-Server Access ```bash\\n# Generate one key\\nssh generate-key server01.example.com --ttl 1hr # Use the same private key for multiple servers (if you have provisioning access)\\n# Note: Currently each key is server-specific, multi-server support coming soon\\n```plaintext ## Command Reference ### ssh generate-key Generate a new temporal SSH key. **Syntax**: ```bash\\nssh generate-key  [options]\\n```plaintext **Options**: - `--user `: SSH user (default: root)\\n- `--ttl `: Key lifetime (default: 1hr)\\n- `--type `: Key type (default: dynamic)\\n- `--ip `: Allowed IP (OTP mode only)\\n- `--principal `: Principal (CA mode only) **Examples**: ```bash\\n# Basic usage\\nssh generate-key server.example.com # Custom user and TTL\\nssh generate-key server.example.com --user deploy --ttl 30min # Vault CA mode\\nssh generate-key server.example.com --type ca --principal admin\\n```plaintext ### ssh deploy-key Deploy a generated key to the target server. **Syntax**: ```bash\\nssh deploy-key \\n```plaintext **Example**: ```bash\\nssh deploy-key abc-123-def-456\\n```plaintext ### ssh list-keys List all active SSH keys. **Syntax**: ```bash\\nssh list-keys [--expired]\\n```plaintext **Examples**: ```bash\\n# List active keys\\nssh list-keys # Show only deployed keys\\nssh list-keys | where deployed == true # Include expired keys\\nssh list-keys --expired\\n```plaintext ### ssh get-key Get detailed information about a specific key. **Syntax**: ```bash\\nssh get-key \\n```plaintext **Example**: ```bash\\nssh get-key abc-123-def-456\\n```plaintext ### ssh revoke-key Immediately revoke a key (removes from server and tracking). **Syntax**: ```bash\\nssh revoke-key \\n```plaintext **Example**: ```bash\\nssh revoke-key abc-123-def-456\\n```plaintext ### ssh connect Auto-generate, deploy, connect, and revoke (all-in-one). **Syntax**: ```bash\\nssh connect  [options]\\n```plaintext **Options**: - `--user `: SSH user (default: root)\\n- `--ttl `: Key lifetime (default: 1hr)\\n- `--type `: Key type (default: dynamic)\\n- `--keep`: Don\'t revoke after disconnect **Examples**: ```bash\\n# Quick connection\\nssh connect server.example.com # Custom user\\nssh connect server.example.com --user deploy # Keep key active after disconnect\\nssh connect server.example.com --keep\\n```plaintext ### ssh stats Show SSH key statistics. **Syntax**: ```bash\\nssh stats\\n```plaintext **Example Output**: ```plaintext\\nSSH Key Statistics: Total generated: 42 Active keys: 10 Expired keys: 32 Keys by type: dynamic: 35 otp: 5 certificate: 2 Last cleanup: 2024-01-01T12:00:00Z Cleaned keys: 5\\n```plaintext ### ssh cleanup Manually trigger cleanup of expired keys. **Syntax**: ```bash\\nssh cleanup\\n```plaintext ### ssh test Run a quick test of the SSH key system. **Syntax**: ```bash\\nssh test  [--user ]\\n```plaintext **Example**: ```bash\\nssh test server.example.com --user root\\n```plaintext ### ssh help Show help information. **Syntax**: ```bash\\nssh help\\n```plaintext ## Duration Formats The `--ttl` option accepts various duration formats: | Format | Example | Meaning |\\n|--------|---------|---------|\\n| Minutes | `30min` | 30 minutes |\\n| Hours | `2hr` | 2 hours |\\n| Mixed | `1hr 30min` | 1.5 hours |\\n| Seconds | `3600sec` | 1 hour | ## Working with Private Keys ### Saving Private Keys When you generate a key, save the private key immediately: ```bash\\n# Generate and save to file\\nssh generate-key server.example.com | get private_key | save -f ~/.ssh/temp_key\\nchmod 600 ~/.ssh/temp_key # Use the key\\nssh -i ~/.ssh/temp_key root@server.example.com # Cleanup\\nrm ~/.ssh/temp_key\\n```plaintext ### Using SSH Agent Add the temporary key to your SSH agent: ```bash\\n# Generate key and extract private key\\nssh generate-key server.example.com | get private_key | save -f /tmp/temp_key\\nchmod 600 /tmp/temp_key # Add to agent\\nssh-add /tmp/temp_key # Connect (agent provides the key automatically)\\nssh root@server.example.com # Remove from agent\\nssh-add -d /tmp/temp_key\\nrm /tmp/temp_key\\n```plaintext ## Troubleshooting ### Key Deployment Fails **Problem**: `ssh deploy-key` returns error **Solutions**: 1. Check SSH connectivity to server: ```bash ssh root@server.example.com Verify provisioning key is configured: echo $PROVISIONING_SSH_KEY Check server SSH daemon: ssh root@server.example.com \\"systemctl status sshd\\"","breadcrumbs":"SSH Temporal Keys User Guide » Generate and Connect with Temporary Key","id":"1606","title":"Generate and Connect with Temporary Key"},"1607":{"body":"Problem : SSH connection fails with \\"Permission denied (publickey)\\" Solutions : Verify key was deployed: ssh list-keys | where id == \\"\\" Check key hasn\'t expired: ssh get-key  | get expires_at Verify private key permissions: chmod 600 /path/to/private/key","breadcrumbs":"SSH Temporal Keys User Guide » Private Key Not Working","id":"1607","title":"Private Key Not Working"},"1608":{"body":"Problem : Expired keys not being removed Solutions : Check orchestrator is running: curl http://localhost:9090/health Trigger manual cleanup: ssh cleanup Check orchestrator logs: tail -f ./data/orchestrator.log | grep SSH","breadcrumbs":"SSH Temporal Keys User Guide » Cleanup Not Running","id":"1608","title":"Cleanup Not Running"},"1609":{"body":"","breadcrumbs":"SSH Temporal Keys User Guide » Best Practices","id":"1609","title":"Best Practices"},"161":{"body":"Create a basic infrastructure configuration: # Generate infrastructure template\\nprovisioning generate infra --new my-infra # This creates: workspace/infra/my-infra/\\n# - config.toml (infrastructure settings)\\n# - settings.k (KCL configuration)","breadcrumbs":"First Deployment » Step 1: Configure Infrastructure","id":"161","title":"Step 1: Configure Infrastructure"},"1610":{"body":"Short TTLs : Use the shortest TTL that works for your task ssh connect server.example.com --ttl 30min Immediate Revocation : Revoke keys when you\'re done ssh revoke-key  Private Key Handling : Never share or commit private keys # Save to temp location, delete after use\\nssh generate-key server.example.com | get private_key | save -f /tmp/key\\n# ... use key ...\\nrm /tmp/key","breadcrumbs":"SSH Temporal Keys User Guide » Security","id":"1610","title":"Security"},"1611":{"body":"Automated Deployments : Generate key in CI/CD #!/bin/bash\\nKEY_ID=$(ssh generate-key prod.example.com --ttl 1hr | get id)\\nssh deploy-key $KEY_ID\\n# Run deployment\\nansible-playbook deploy.yml\\nssh revoke-key $KEY_ID Interactive Use : Use ssh connect for quick access ssh connect dev.example.com Monitoring : Check statistics regularly ssh stats","breadcrumbs":"SSH Temporal Keys User Guide » Workflow Integration","id":"1611","title":"Workflow Integration"},"1612":{"body":"","breadcrumbs":"SSH Temporal Keys User Guide » Advanced Usage","id":"1612","title":"Advanced Usage"},"1613":{"body":"If your organization uses HashiCorp Vault: CA Mode (Recommended) # Generate CA-signed certificate\\nssh generate-key server.example.com --type ca --principal admin --ttl 1hr # Vault signs your public key\\n# Server must trust Vault CA certificate\\n```plaintext **Setup** (one-time): ```bash\\n# On servers, add to /etc/ssh/sshd_config:\\nTrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem # Get Vault CA public key:\\nvault read -field=public_key ssh/config/ca | \\\\ sudo tee /etc/ssh/trusted-user-ca-keys.pem # Restart SSH:\\nsudo systemctl restart sshd\\n```plaintext #### OTP Mode ```bash\\n# Generate one-time password\\nssh generate-key server.example.com --type otp --ip 192.168.1.100 # Use the OTP to connect (single use only)\\n```plaintext ### Scripting Use in scripts for automated operations: ```nushell\\n# deploy.nu\\ndef deploy [target: string] { let key = (ssh generate-key $target --ttl 1hr) ssh deploy-key $key.id # Run deployment try { ssh $\\"root@($target)\\" \\"bash /path/to/deploy.sh\\" } catch { print \\"Deployment failed\\" } # Always cleanup ssh revoke-key $key.id\\n}\\n```plaintext ## API Integration For programmatic access, use the REST API: ```bash\\n# Generate key\\ncurl -X POST http://localhost:9090/api/v1/ssh/generate \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{ \\"key_type\\": \\"dynamickeypair\\", \\"user\\": \\"root\\", \\"target_server\\": \\"server.example.com\\", \\"ttl_seconds\\": 3600 }\' # Deploy key\\ncurl -X POST http://localhost:9090/api/v1/ssh/{key_id}/deploy # List keys\\ncurl http://localhost:9090/api/v1/ssh/keys # Get stats\\ncurl http://localhost:9090/api/v1/ssh/stats\\n```plaintext ## FAQ **Q: Can I use the same key for multiple servers?**\\nA: Currently, each key is tied to a specific server. Multi-server support is planned. **Q: What happens if the orchestrator crashes?**\\nA: Keys in memory are lost, but keys already deployed to servers remain until their expiration time. **Q: Can I extend the TTL of an existing key?**\\nA: No, you must generate a new key. This is by design for security. **Q: What\'s the maximum TTL?**\\nA: Configurable by admin, default maximum is 24 hours. **Q: Are private keys stored anywhere?**\\nA: Private keys exist only in memory during generation and are shown once to the user. They are never written to disk by the system. **Q: What happens if cleanup fails?**\\nA: The key remains in authorized_keys until the next cleanup run. You can trigger manual cleanup with `ssh cleanup`. **Q: Can I use this with non-root users?**\\nA: Yes, use `--user ` when generating the key. **Q: How do I know when my key will expire?**\\nA: Use `ssh get-key ` to see the exact expiration timestamp. ## Support For issues or questions: 1. Check orchestrator logs: `tail -f ./data/orchestrator.log`\\n2. Run diagnostics: `ssh stats`\\n3. Test connectivity: `ssh test server.example.com`\\n4. Review documentation: `SSH_KEY_MANAGEMENT.md` ## See Also - **Architecture**: `SSH_KEY_MANAGEMENT.md`\\n- **Implementation**: `SSH_IMPLEMENTATION_SUMMARY.md`\\n- **Configuration**: `config/ssh-config.toml.example`","breadcrumbs":"SSH Temporal Keys User Guide » Vault Integration","id":"1613","title":"Vault Integration"},"1614":{"body":"Version : 1.0.0 Last Updated : 2025-10-09 Target Audience : Developers, DevOps Engineers, System Administrators","breadcrumbs":"Plugin Integration Guide » Nushell Plugin Integration Guide","id":"1614","title":"Nushell Plugin Integration Guide"},"1615":{"body":"Overview Why Native Plugins? Prerequisites Installation Quick Start (5 Minutes) Authentication Plugin (nu_plugin_auth) KMS Plugin (nu_plugin_kms) Orchestrator Plugin (nu_plugin_orchestrator) Integration Examples Best Practices Troubleshooting Migration Guide Advanced Configuration Security Considerations FAQ","breadcrumbs":"Plugin Integration Guide » Table of Contents","id":"1615","title":"Table of Contents"},"1616":{"body":"The Provisioning Platform provides three native Nushell plugins that dramatically improve performance and user experience compared to traditional HTTP API calls: Plugin Purpose Performance Gain nu_plugin_auth JWT authentication, MFA, session management 20% faster nu_plugin_kms Encryption/decryption with multiple KMS backends 10x faster nu_plugin_orchestrator Orchestrator operations without HTTP overhead 50x faster","breadcrumbs":"Plugin Integration Guide » Overview","id":"1616","title":"Overview"},"1617":{"body":"Traditional HTTP Flow:\\nUser Command → HTTP Request → Network → Server Processing → Response → Parse JSON Total: ~50-100ms per operation Plugin Flow:\\nUser Command → Direct Rust Function Call → Return Nushell Data Structure Total: ~1-10ms per operation\\n```plaintext ### Key Features ✅ **Performance**: 10-50x faster than HTTP API\\n✅ **Type Safety**: Full Nushell type system integration\\n✅ **Pipeline Support**: Native Nushell data structures\\n✅ **Offline Capability**: KMS and orchestrator work without network\\n✅ **OS Integration**: Native keyring for secure token storage\\n✅ **Graceful Fallback**: HTTP still available if plugins not installed --- ## Why Native Plugins? ### Performance Comparison Real-world benchmarks from production workload: | Operation | HTTP API | Plugin | Improvement | Speedup |\\n|-----------|----------|--------|-------------|---------|\\n| **KMS Encrypt (RustyVault)** | ~50ms | ~5ms | -45ms | **10x** |\\n| **KMS Decrypt (RustyVault)** | ~50ms | ~5ms | -45ms | **10x** |\\n| **KMS Encrypt (Age)** | ~30ms | ~3ms | -27ms | **10x** |\\n| **KMS Decrypt (Age)** | ~30ms | ~3ms | -27ms | **10x** |\\n| **Orchestrator Status** | ~30ms | ~1ms | -29ms | **30x** |\\n| **Orchestrator Tasks List** | ~50ms | ~5ms | -45ms | **10x** |\\n| **Orchestrator Validate** | ~100ms | ~10ms | -90ms | **10x** |\\n| **Auth Login** | ~100ms | ~80ms | -20ms | 1.25x |\\n| **Auth Verify** | ~50ms | ~10ms | -40ms | **5x** |\\n| **Auth MFA Verify** | ~80ms | ~60ms | -20ms | 1.3x | ### Use Case: Batch Processing **Scenario**: Encrypt 100 configuration files ```nushell\\n# HTTP API approach\\nls configs/*.yaml | each { |file| http post http://localhost:9998/encrypt { data: (open $file) }\\n} | save encrypted/\\n# Total time: ~5 seconds (50ms × 100) # Plugin approach\\nls configs/*.yaml | each { |file| kms encrypt (open $file) --backend rustyvault\\n} | save encrypted/\\n# Total time: ~0.5 seconds (5ms × 100)\\n# Result: 10x faster\\n```plaintext ### Developer Experience Benefits **1. Native Nushell Integration** ```nushell\\n# HTTP: Parse JSON, check status codes\\nlet result = http post http://localhost:9998/encrypt { data: \\"secret\\" }\\nif $result.status == \\"success\\" { $result.encrypted\\n} else { error make { msg: $result.error }\\n} # Plugin: Direct return values\\nkms encrypt \\"secret\\"\\n# Returns encrypted string directly, errors use Nushell\'s error system\\n```plaintext **2. Pipeline Friendly** ```nushell\\n# HTTP: Requires wrapping, JSON parsing\\n[\\"secret1\\", \\"secret2\\"] | each { |s| (http post http://localhost:9998/encrypt { data: $s }).encrypted\\n} # Plugin: Natural pipeline flow\\n[\\"secret1\\", \\"secret2\\"] | each { |s| kms encrypt $s }\\n```plaintext **3. Tab Completion** ```nushell\\n# All plugin commands have full tab completion\\nkms \\n# → encrypt, decrypt, generate-key, status, backends kms encrypt --\\n# → --backend, --key, --context\\n```plaintext --- ## Prerequisites ### Required Software | Software | Minimum Version | Purpose |\\n|----------|----------------|---------|\\n| **Nushell** | 0.107.1 | Shell and plugin runtime |\\n| **Rust** | 1.75+ | Building plugins from source |\\n| **Cargo** | (included with Rust) | Build tool | ### Optional Dependencies | Software | Purpose | Platform |\\n|----------|---------|----------|\\n| **gnome-keyring** | Secure token storage | Linux |\\n| **kwallet** | Secure token storage | Linux (KDE) |\\n| **age** | Age encryption backend | All |\\n| **RustyVault** | High-performance KMS | All | ### Platform Support | Platform | Status | Notes |\\n|----------|--------|-------|\\n| **macOS** | ✅ Full | Keychain integration |\\n| **Linux** | ✅ Full | Requires keyring service |\\n| **Windows** | ✅ Full | Credential Manager integration |\\n| **FreeBSD** | ⚠️ Partial | No keyring integration | --- ## Installation ### Step 1: Clone or Navigate to Plugin Directory ```bash\\ncd /Users/Akasha/project-provisioning/provisioning/core/plugins/nushell-plugins\\n```plaintext ### Step 2: Build All Plugins ```bash\\n# Build in release mode (optimized for performance)\\ncargo build --release --all # Or build individually\\ncargo build --release -p nu_plugin_auth\\ncargo build --release -p nu_plugin_kms\\ncargo build --release -p nu_plugin_orchestrator\\n```plaintext **Expected output:** ```plaintext Compiling nu_plugin_auth v0.1.0 Compiling nu_plugin_kms v0.1.0 Compiling nu_plugin_orchestrator v0.1.0 Finished release [optimized] target(s) in 2m 15s\\n```plaintext ### Step 3: Register Plugins with Nushell ```bash\\n# Register all three plugins\\nplugin add target/release/nu_plugin_auth\\nplugin add target/release/nu_plugin_kms\\nplugin add target/release/nu_plugin_orchestrator # On macOS, full paths:\\nplugin add $PWD/target/release/nu_plugin_auth\\nplugin add $PWD/target/release/nu_plugin_kms\\nplugin add $PWD/target/release/nu_plugin_orchestrator\\n```plaintext ### Step 4: Verify Installation ```bash\\n# List registered plugins\\nplugin list | where name =~ \\"auth|kms|orch\\" # Test each plugin\\nauth --help\\nkms --help\\norch --help\\n```plaintext **Expected output:** ```plaintext\\n╭───┬─────────────────────────┬─────────┬───────────────────────────────────╮\\n│ # │ name │ version │ filename │\\n├───┼─────────────────────────┼─────────┼───────────────────────────────────┤\\n│ 0 │ nu_plugin_auth │ 0.1.0 │ .../nu_plugin_auth │\\n│ 1 │ nu_plugin_kms │ 0.1.0 │ .../nu_plugin_kms │\\n│ 2 │ nu_plugin_orchestrator │ 0.1.0 │ .../nu_plugin_orchestrator │\\n╰───┴─────────────────────────┴─────────┴───────────────────────────────────╯\\n```plaintext ### Step 5: Configure Environment (Optional) ```bash\\n# Add to ~/.config/nushell/env.nu\\n$env.RUSTYVAULT_ADDR = \\"http://localhost:8200\\"\\n$env.RUSTYVAULT_TOKEN = \\"your-vault-token\\"\\n$env.CONTROL_CENTER_URL = \\"http://localhost:3000\\"\\n$env.ORCHESTRATOR_DATA_DIR = \\"/opt/orchestrator/data\\"\\n```plaintext --- ## Quick Start (5 Minutes) ### 1. Authentication Workflow ```nushell\\n# Login (password prompted securely)\\nauth login admin\\n# ✓ Login successful\\n# User: admin\\n# Role: Admin\\n# Expires: 2025-10-09T14:30:00Z # Verify session\\nauth verify\\n# {\\n# \\"active\\": true,\\n# \\"user\\": \\"admin\\",\\n# \\"role\\": \\"Admin\\",\\n# \\"expires_at\\": \\"2025-10-09T14:30:00Z\\"\\n# } # Enroll in MFA (optional but recommended)\\nauth mfa enroll totp\\n# QR code displayed, save backup codes # Verify MFA\\nauth mfa verify --code 123456\\n# ✓ MFA verification successful # Logout\\nauth logout\\n# ✓ Logged out successfully\\n```plaintext ### 2. KMS Operations ```nushell\\n# Encrypt data\\nkms encrypt \\"my secret data\\"\\n# vault:v1:8GawgGuP... # Decrypt data\\nkms decrypt \\"vault:v1:8GawgGuP...\\"\\n# my secret data # Check available backends\\nkms status\\n# {\\n# \\"backend\\": \\"rustyvault\\",\\n# \\"status\\": \\"healthy\\",\\n# \\"url\\": \\"http://localhost:8200\\"\\n# } # Encrypt with specific backend\\nkms encrypt \\"data\\" --backend age --key age1xxxxxxx\\n```plaintext ### 3. Orchestrator Operations ```nushell\\n# Check orchestrator status (no HTTP call)\\norch status\\n# {\\n# \\"active_tasks\\": 5,\\n# \\"completed_tasks\\": 120,\\n# \\"health\\": \\"healthy\\"\\n# } # Validate workflow\\norch validate workflows/deploy.k\\n# {\\n# \\"valid\\": true,\\n# \\"workflow\\": { \\"name\\": \\"deploy_k8s\\", \\"operations\\": 5 }\\n# } # List running tasks\\norch tasks --status running\\n# [ { \\"task_id\\": \\"task_123\\", \\"name\\": \\"deploy_k8s\\", \\"progress\\": 45 } ]\\n```plaintext ### 4. Combined Workflow ```nushell\\n# Complete authenticated deployment pipeline\\nauth login admin | if $in.success { auth verify } | if $in.active { orch validate workflows/production.k | if $in.valid { kms encrypt (open secrets.yaml | to json) | save production-secrets.enc } }\\n# ✓ Pipeline completed successfully\\n```plaintext --- ## Authentication Plugin (nu_plugin_auth) The authentication plugin manages JWT-based authentication, MFA enrollment/verification, and session management with OS-native keyring integration. ### Available Commands | Command | Purpose | Example |\\n|---------|---------|---------|\\n| `auth login` | Login and store JWT | `auth login admin` |\\n| `auth logout` | Logout and clear tokens | `auth logout` |\\n| `auth verify` | Verify current session | `auth verify` |\\n| `auth sessions` | List active sessions | `auth sessions` |\\n| `auth mfa enroll` | Enroll in MFA | `auth mfa enroll totp` |\\n| `auth mfa verify` | Verify MFA code | `auth mfa verify --code 123456` | ### Command Reference #### `auth login  [password]` Login to provisioning platform and store JWT tokens securely in OS keyring. **Arguments:** - `username` (required): Username for authentication\\n- `password` (optional): Password (prompted if not provided) **Flags:** - `--url `: Control center URL (default: `http://localhost:3000`)\\n- `--password `: Password (alternative to positional argument) **Examples:** ```nushell\\n# Interactive password prompt (recommended)\\nauth login admin\\n# Password: ••••••••\\n# ✓ Login successful\\n# User: admin\\n# Role: Admin\\n# Expires: 2025-10-09T14:30:00Z # Password in command (not recommended for production)\\nauth login admin mypassword # Custom control center URL\\nauth login admin --url https://control-center.example.com # Pipeline usage\\nlet creds = { username: \\"admin\\", password: (input --suppress-output \\"Password: \\") }\\nauth login $creds.username $creds.password\\n```plaintext **Token Storage Locations:** - **macOS**: Keychain Access (`login` keychain)\\n- **Linux**: Secret Service API (gnome-keyring, kwallet)\\n- **Windows**: Windows Credential Manager **Security Notes:** - Tokens encrypted at rest by OS\\n- Requires user authentication to access (macOS Touch ID, Linux password)\\n- Never stored in plain text files #### `auth logout` Logout from current session and remove stored tokens from keyring. **Examples:** ```nushell\\n# Simple logout\\nauth logout\\n# ✓ Logged out successfully # Conditional logout\\nif (auth verify | get active) { auth logout echo \\"Session terminated\\"\\n} # Logout all sessions (requires admin role)\\nauth sessions | each { |sess| auth logout --session-id $sess.session_id\\n}\\n```plaintext #### `auth verify` Verify current session status and check token validity. **Returns:** - `active` (bool): Whether session is active\\n- `user` (string): Username\\n- `role` (string): User role\\n- `expires_at` (datetime): Token expiration\\n- `mfa_verified` (bool): MFA verification status **Examples:** ```nushell\\n# Check if logged in\\nauth verify\\n# {\\n# \\"active\\": true,\\n# \\"user\\": \\"admin\\",\\n# \\"role\\": \\"Admin\\",\\n# \\"expires_at\\": \\"2025-10-09T14:30:00Z\\",\\n# \\"mfa_verified\\": true\\n# } # Pipeline usage\\nif (auth verify | get active) { echo \\"✓ Authenticated\\"\\n} else { auth login admin\\n} # Check expiration\\nlet session = auth verify\\nif ($session.expires_at | into datetime) < (date now) { echo \\"Session expired, re-authenticating...\\" auth login $session.user\\n}\\n```plaintext #### `auth sessions` List all active sessions for current user. **Examples:** ```nushell\\n# List all sessions\\nauth sessions\\n# [\\n# {\\n# \\"session_id\\": \\"sess_abc123\\",\\n# \\"created_at\\": \\"2025-10-09T12:00:00Z\\",\\n# \\"expires_at\\": \\"2025-10-09T14:30:00Z\\",\\n# \\"ip_address\\": \\"192.168.1.100\\",\\n# \\"user_agent\\": \\"nushell/0.107.1\\"\\n# }\\n# ] # Filter recent sessions (last hour)\\nauth sessions | where created_at > ((date now) - 1hr) # Find sessions by IP\\nauth sessions | where ip_address =~ \\"192.168\\" # Count active sessions\\nauth sessions | length\\n```plaintext #### `auth mfa enroll ` Enroll in Multi-Factor Authentication (TOTP or WebAuthn). **Arguments:** - `type` (required): MFA type (`totp` or `webauthn`) **TOTP Enrollment:** ```nushell\\nauth mfa enroll totp\\n# ✓ TOTP enrollment initiated\\n#\\n# Scan this QR code with your authenticator app:\\n#\\n# ████ ▄▄▄▄▄ █▀█ █▄▀▀▀▄ ▄▄▄▄▄ ████\\n# ████ █ █ █▀▀▀█▄ ▀▀█ █ █ ████\\n# ████ █▄▄▄█ █ █▀▄ ▀▄▄█ █▄▄▄█ ████\\n# (QR code continues...)\\n#\\n# Or enter manually:\\n# Secret: JBSWY3DPEHPK3PXP\\n# URL: otpauth://totp/Provisioning:admin?secret=JBSWY3DPEHPK3PXP&issuer=Provisioning\\n#\\n# Backup codes (save securely):\\n# 1. ABCD-EFGH-IJKL\\n# 2. MNOP-QRST-UVWX\\n# 3. YZAB-CDEF-GHIJ\\n# (8 more codes...)\\n```plaintext **WebAuthn Enrollment:** ```nushell\\nauth mfa enroll webauthn\\n# ✓ WebAuthn enrollment initiated\\n#\\n# Insert your security key and touch the button...\\n# (waiting for device interaction)\\n#\\n# ✓ Security key registered successfully\\n# Device: YubiKey 5 NFC\\n# Created: 2025-10-09T13:00:00Z\\n```plaintext **Supported Authenticator Apps:** - Google Authenticator\\n- Microsoft Authenticator\\n- Authy\\n- 1Password\\n- Bitwarden **Supported Hardware Keys:** - YubiKey (all models)\\n- Titan Security Key\\n- Feitian ePass\\n- macOS Touch ID\\n- Windows Hello #### `auth mfa verify --code ` Verify MFA code (TOTP or backup code). **Flags:** - `--code ` (required): 6-digit TOTP code or backup code **Examples:** ```nushell\\n# Verify TOTP code\\nauth mfa verify --code 123456\\n# ✓ MFA verification successful # Verify backup code\\nauth mfa verify --code ABCD-EFGH-IJKL\\n# ✓ MFA verification successful (backup code used)\\n# Warning: This backup code cannot be used again # Pipeline usage\\nlet code = input \\"MFA code: \\"\\nauth mfa verify --code $code\\n```plaintext **Error Cases:** ```nushell\\n# Invalid code\\nauth mfa verify --code 999999\\n# Error: Invalid MFA code\\n# → Verify time synchronization on your device # Rate limited\\nauth mfa verify --code 123456\\n# Error: Too many failed attempts\\n# → Wait 5 minutes before trying again # No MFA enrolled\\nauth mfa verify --code 123456\\n# Error: MFA not enrolled for this user\\n# → Run: auth mfa enroll totp\\n```plaintext ### Environment Variables | Variable | Description | Default |\\n|----------|-------------|---------|\\n| `USER` | Default username | Current OS user |\\n| `CONTROL_CENTER_URL` | Control center URL | `http://localhost:3000` |\\n| `AUTH_KEYRING_SERVICE` | Keyring service name | `provisioning-auth` | ### Troubleshooting Authentication **\\"No active session\\"** ```nushell\\n# Solution: Login first\\nauth login \\n```plaintext **\\"Keyring error\\" (macOS)** ```bash\\n# Check Keychain Access permissions\\n# System Preferences → Security & Privacy → Privacy → Full Disk Access\\n# Add: /Applications/Nushell.app (or /usr/local/bin/nu) # Or grant access manually\\nsecurity unlock-keychain ~/Library/Keychains/login.keychain-db\\n```plaintext **\\"Keyring error\\" (Linux)** ```bash\\n# Install keyring service\\nsudo apt install gnome-keyring # Ubuntu/Debian\\nsudo dnf install gnome-keyring # Fedora\\nsudo pacman -S gnome-keyring # Arch # Or use KWallet (KDE)\\nsudo apt install kwalletmanager # Start keyring daemon\\neval $(gnome-keyring-daemon --start)\\nexport $(gnome-keyring-daemon --start --components=secrets)\\n```plaintext **\\"MFA verification failed\\"** ```nushell\\n# Check time synchronization (TOTP requires accurate time)\\n# macOS:\\nsudo sntp -sS time.apple.com # Linux:\\nsudo ntpdate pool.ntp.org\\n# Or\\nsudo systemctl restart systemd-timesyncd # Use backup code if TOTP not working\\nauth mfa verify --code ABCD-EFGH-IJKL\\n```plaintext --- ## KMS Plugin (nu_plugin_kms) The KMS plugin provides high-performance encryption and decryption using multiple backend providers. ### Supported Backends | Backend | Performance | Use Case | Setup Complexity |\\n|---------|------------|----------|------------------|\\n| **rustyvault** | ⚡ Very Fast (~5ms) | Production KMS | Medium |\\n| **age** | ⚡ Very Fast (~3ms) | Local development | Low |\\n| **cosmian** | 🐢 Moderate (~30ms) | Cloud KMS | Medium |\\n| **aws** | 🐢 Moderate (~50ms) | AWS environments | Medium |\\n| **vault** | 🐢 Moderate (~40ms) | Enterprise KMS | High | ### Backend Selection Guide **Choose `rustyvault` when:** - ✅ Running in production with high throughput requirements\\n- ✅ Need ~5ms encryption/decryption latency\\n- ✅ Have RustyVault server deployed\\n- ✅ Require key rotation and versioning **Choose `age` when:** - ✅ Developing locally without external dependencies\\n- ✅ Need simple file encryption\\n- ✅ Want ~3ms latency\\n- ❌ Don\'t need centralized key management **Choose `cosmian` when:** - ✅ Using Cosmian KMS service\\n- ✅ Need cloud-based key management\\n- ⚠️ Can accept ~30ms latency **Choose `aws` when:** - ✅ Deployed on AWS infrastructure\\n- ✅ Using AWS IAM for access control\\n- ✅ Need AWS KMS integration\\n- ⚠️ Can accept ~50ms latency **Choose `vault` when:** - ✅ Using HashiCorp Vault enterprise\\n- ✅ Need advanced policy management\\n- ✅ Require audit trails\\n- ⚠️ Can accept ~40ms latency ### Available Commands | Command | Purpose | Example |\\n|---------|---------|---------|\\n| `kms encrypt` | Encrypt data | `kms encrypt \\"secret\\"` |\\n| `kms decrypt` | Decrypt data | `kms decrypt \\"vault:v1:...\\"` |\\n| `kms generate-key` | Generate DEK | `kms generate-key --spec AES256` |\\n| `kms status` | Backend status | `kms status` | ### Command Reference #### `kms encrypt  [--backend ]` Encrypt data using specified KMS backend. **Arguments:** - `data` (required): Data to encrypt (string or binary) **Flags:** - `--backend `: KMS backend (`rustyvault`, `age`, `cosmian`, `aws`, `vault`)\\n- `--key `: Key ID or recipient (backend-specific)\\n- `--context `: Additional authenticated data (AAD) **Examples:** ```nushell\\n# Auto-detect backend from environment\\nkms encrypt \\"secret configuration data\\"\\n# vault:v1:8GawgGuP+emDKX5q... # RustyVault backend\\nkms encrypt \\"data\\" --backend rustyvault --key provisioning-main\\n# vault:v1:abc123def456... # Age backend (local encryption)\\nkms encrypt \\"data\\" --backend age --key age1xxxxxxxxx\\n# -----BEGIN AGE ENCRYPTED FILE-----\\n# YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+...\\n# -----END AGE ENCRYPTED FILE----- # AWS KMS\\nkms encrypt \\"data\\" --backend aws --key alias/provisioning\\n# AQICAHhwbGF0Zm9ybS1wcm92aXNpb25p... # With context (AAD for additional security)\\nkms encrypt \\"data\\" --backend rustyvault --key provisioning-main --context \\"user=admin,env=production\\" # Encrypt file contents\\nkms encrypt (open config.yaml) --backend rustyvault | save config.yaml.enc # Encrypt multiple files\\nls configs/*.yaml | each { |file| kms encrypt (open $file.name) --backend age | save $\\"encrypted/($file.name).enc\\"\\n}\\n```plaintext **Output Formats:** - **RustyVault**: `vault:v1:base64_ciphertext`\\n- **Age**: `-----BEGIN AGE ENCRYPTED FILE-----...-----END AGE ENCRYPTED FILE-----`\\n- **AWS**: `base64_aws_kms_ciphertext`\\n- **Cosmian**: `cosmian:v1:base64_ciphertext` #### `kms decrypt  [--backend ]` Decrypt KMS-encrypted data. **Arguments:** - `encrypted` (required): Encrypted data (detects format automatically) **Flags:** - `--backend `: KMS backend (auto-detected from format if not specified)\\n- `--context `: Additional authenticated data (must match encryption context) **Examples:** ```nushell\\n# Auto-detect backend from format\\nkms decrypt \\"vault:v1:8GawgGuP...\\"\\n# secret configuration data # Explicit backend\\nkms decrypt \\"vault:v1:abc123...\\" --backend rustyvault # Age decryption\\nkms decrypt \\"-----BEGIN AGE ENCRYPTED FILE-----...\\"\\n# (uses AGE_IDENTITY from environment) # With context (must match encryption context)\\nkms decrypt \\"vault:v1:abc123...\\" --context \\"user=admin,env=production\\" # Decrypt file\\nkms decrypt (open config.yaml.enc) | save config.yaml # Decrypt multiple files\\nls encrypted/*.enc | each { |file| kms decrypt (open $file.name) | save $\\"configs/(($file.name | path basename) | str replace \'.enc\' \'\')\\"\\n} # Pipeline decryption\\nopen secrets.json | get database_password_enc | kms decrypt | str trim | psql --dbname mydb --password\\n```plaintext **Error Cases:** ```nushell\\n# Invalid ciphertext\\nkms decrypt \\"invalid_data\\"\\n# Error: Invalid ciphertext format\\n# → Verify data was encrypted with KMS # Context mismatch\\nkms decrypt \\"vault:v1:abc...\\" --context \\"wrong=context\\"\\n# Error: Authentication failed (AAD mismatch)\\n# → Verify encryption context matches # Backend unavailable\\nkms decrypt \\"vault:v1:abc...\\"\\n# Error: Failed to connect to RustyVault at http://localhost:8200\\n# → Check RustyVault is running: curl http://localhost:8200/v1/sys/health\\n```plaintext #### `kms generate-key [--spec ]` Generate data encryption key (DEK) using KMS envelope encryption. **Flags:** - `--spec `: Key specification (`AES128` or `AES256`, default: `AES256`)\\n- `--backend `: KMS backend **Examples:** ```nushell\\n# Generate AES-256 key\\nkms generate-key\\n# {\\n# \\"plaintext\\": \\"rKz3N8xPq...\\", # base64-encoded key\\n# \\"ciphertext\\": \\"vault:v1:...\\", # encrypted DEK\\n# \\"spec\\": \\"AES256\\"\\n# } # Generate AES-128 key\\nkms generate-key --spec AES128 # Use in envelope encryption pattern\\nlet dek = kms generate-key\\nlet encrypted_data = ($data | openssl enc -aes-256-cbc -K $dek.plaintext)\\n{ data: $encrypted_data, encrypted_key: $dek.ciphertext\\n} | save secure_data.json # Later, decrypt:\\nlet envelope = open secure_data.json\\nlet dek = kms decrypt $envelope.encrypted_key\\n$envelope.data | openssl enc -d -aes-256-cbc -K $dek\\n```plaintext **Use Cases:** - Envelope encryption (encrypt large data locally, protect DEK with KMS)\\n- Database field encryption\\n- File encryption with key wrapping #### `kms status` Show KMS backend status, configuration, and health. **Examples:** ```nushell\\n# Show current backend status\\nkms status\\n# {\\n# \\"backend\\": \\"rustyvault\\",\\n# \\"status\\": \\"healthy\\",\\n# \\"url\\": \\"http://localhost:8200\\",\\n# \\"mount_point\\": \\"transit\\",\\n# \\"version\\": \\"0.1.0\\",\\n# \\"latency_ms\\": 5\\n# } # Check all configured backends\\nkms status --all\\n# [\\n# { \\"backend\\": \\"rustyvault\\", \\"status\\": \\"healthy\\", ... },\\n# { \\"backend\\": \\"age\\", \\"status\\": \\"available\\", ... },\\n# { \\"backend\\": \\"aws\\", \\"status\\": \\"unavailable\\", \\"error\\": \\"...\\" }\\n# ] # Filter to specific backend\\nkms status | where backend == \\"rustyvault\\" # Health check in automation\\nif (kms status | get status) == \\"healthy\\" { echo \\"✓ KMS operational\\"\\n} else { error make { msg: \\"KMS unhealthy\\" }\\n}\\n```plaintext ### Backend Configuration #### RustyVault Backend ```bash\\n# Environment variables\\nexport RUSTYVAULT_ADDR=\\"http://localhost:8200\\"\\nexport RUSTYVAULT_TOKEN=\\"hvs.xxxxxxxxxxxxx\\"\\nexport RUSTYVAULT_MOUNT=\\"transit\\" # Transit engine mount point\\nexport RUSTYVAULT_KEY=\\"provisioning-main\\" # Default key name\\n```plaintext ```nushell\\n# Usage\\nkms encrypt \\"data\\" --backend rustyvault --key provisioning-main\\n```plaintext **Setup RustyVault:** ```bash\\n# Start RustyVault\\nrustyvault server -dev # Enable transit engine\\nrustyvault secrets enable transit # Create encryption key\\nrustyvault write -f transit/keys/provisioning-main\\n```plaintext #### Age Backend ```bash\\n# Generate Age keypair\\nage-keygen -o ~/.age/key.txt # Environment variables\\nexport AGE_IDENTITY=\\"$HOME/.age/key.txt\\" # Private key\\nexport AGE_RECIPIENT=\\"age1xxxxxxxxx\\" # Public key (from key.txt)\\n```plaintext ```nushell\\n# Usage\\nkms encrypt \\"data\\" --backend age\\nkms decrypt (open file.enc) --backend age\\n```plaintext #### AWS KMS Backend ```bash\\n# AWS credentials\\nexport AWS_REGION=\\"us-east-1\\"\\nexport AWS_ACCESS_KEY_ID=\\"AKIAXXXXX\\"\\nexport AWS_SECRET_ACCESS_KEY=\\"xxxxx\\" # KMS configuration\\nexport AWS_KMS_KEY_ID=\\"alias/provisioning\\"\\n```plaintext ```nushell\\n# Usage\\nkms encrypt \\"data\\" --backend aws --key alias/provisioning\\n```plaintext **Setup AWS KMS:** ```bash\\n# Create KMS key\\naws kms create-key --description \\"Provisioning Platform\\" # Create alias\\naws kms create-alias --alias-name alias/provisioning --target-key-id  # Grant permissions\\naws kms create-grant --key-id  --grantee-principal  \\\\ --operations Encrypt Decrypt GenerateDataKey\\n```plaintext #### Cosmian Backend ```bash\\n# Cosmian KMS configuration\\nexport KMS_HTTP_URL=\\"http://localhost:9998\\"\\nexport KMS_HTTP_BACKEND=\\"cosmian\\"\\nexport COSMIAN_API_KEY=\\"your-api-key\\"\\n```plaintext ```nushell\\n# Usage\\nkms encrypt \\"data\\" --backend cosmian\\n```plaintext #### Vault Backend (HashiCorp) ```bash\\n# Vault configuration\\nexport VAULT_ADDR=\\"https://vault.example.com:8200\\"\\nexport VAULT_TOKEN=\\"hvs.xxxxxxxxxxxxx\\"\\nexport VAULT_MOUNT=\\"transit\\"\\nexport VAULT_KEY=\\"provisioning\\"\\n```plaintext ```nushell\\n# Usage\\nkms encrypt \\"data\\" --backend vault --key provisioning\\n```plaintext ### Performance Benchmarks **Test Setup:** - Data size: 1KB\\n- Iterations: 1000\\n- Hardware: Apple M1, 16GB RAM\\n- Network: localhost **Results:** | Backend | Encrypt (avg) | Decrypt (avg) | Throughput (ops/sec) |\\n|---------|---------------|---------------|----------------------|\\n| RustyVault | 4.8ms | 5.1ms | ~200 |\\n| Age | 2.9ms | 3.2ms | ~320 |\\n| Cosmian HTTP | 31ms | 29ms | ~33 |\\n| AWS KMS | 52ms | 48ms | ~20 |\\n| Vault | 38ms | 41ms | ~25 | **Scaling Test (1000 operations):** ```nushell\\n# RustyVault: ~5 seconds\\n0..1000 | each { |_| kms encrypt \\"data\\" --backend rustyvault } | length\\n# Age: ~3 seconds\\n0..1000 | each { |_| kms encrypt \\"data\\" --backend age } | length\\n```plaintext ### Troubleshooting KMS **\\"RustyVault connection failed\\"** ```bash\\n# Check RustyVault is running\\ncurl http://localhost:8200/v1/sys/health\\n# Expected: { \\"initialized\\": true, \\"sealed\\": false } # Check environment\\necho $env.RUSTYVAULT_ADDR\\necho $env.RUSTYVAULT_TOKEN # Test authentication\\ncurl -H \\"X-Vault-Token: $RUSTYVAULT_TOKEN\\" $RUSTYVAULT_ADDR/v1/sys/health\\n```plaintext **\\"Age encryption failed\\"** ```bash\\n# Check Age keys exist\\nls -la ~/.age/\\n# Expected: key.txt # Verify key format\\ncat ~/.age/key.txt | head -1\\n# Expected: # created: \\n# Line 2: # public key: age1xxxxx\\n# Line 3: AGE-SECRET-KEY-xxxxx # Extract public key\\nexport AGE_RECIPIENT=$(grep \\"public key:\\" ~/.age/key.txt | cut -d: -f2 | tr -d \' \')\\necho $AGE_RECIPIENT\\n```plaintext **\\"AWS KMS access denied\\"** ```bash\\n# Verify AWS credentials\\naws sts get-caller-identity\\n# Expected: Account, UserId, Arn # Check KMS key permissions\\naws kms describe-key --key-id alias/provisioning # Test encryption\\naws kms encrypt --key-id alias/provisioning --plaintext \\"test\\"\\n```plaintext --- ## Orchestrator Plugin (nu_plugin_orchestrator) The orchestrator plugin provides direct file-based access to orchestrator state, eliminating HTTP overhead for status queries and validation. ### Available Commands | Command | Purpose | Example |\\n|---------|---------|---------|\\n| `orch status` | Orchestrator status | `orch status` |\\n| `orch validate` | Validate workflow | `orch validate workflow.k` |\\n| `orch tasks` | List tasks | `orch tasks --status running` | ### Command Reference #### `orch status [--data-dir ]` Get orchestrator status from local files (no HTTP, ~1ms latency). **Flags:** - `--data-dir `: Data directory (default from `ORCHESTRATOR_DATA_DIR`) **Examples:** ```nushell\\n# Default data directory\\norch status\\n# {\\n# \\"active_tasks\\": 5,\\n# \\"completed_tasks\\": 120,\\n# \\"failed_tasks\\": 2,\\n# \\"pending_tasks\\": 3,\\n# \\"uptime\\": \\"2d 4h 15m\\",\\n# \\"health\\": \\"healthy\\"\\n# } # Custom data directory\\norch status --data-dir /opt/orchestrator/data # Monitor in loop\\nwhile true { clear orch status | table sleep 5sec\\n} # Alert on failures\\nif (orch status | get failed_tasks) > 0 { echo \\"⚠️ Failed tasks detected!\\"\\n}\\n```plaintext #### `orch validate  [--strict]` Validate workflow KCL file syntax and structure. **Arguments:** - `workflow.k` (required): Path to KCL workflow file **Flags:** - `--strict`: Enable strict validation (warnings as errors) **Examples:** ```nushell\\n# Basic validation\\norch validate workflows/deploy.k\\n# {\\n# \\"valid\\": true,\\n# \\"workflow\\": {\\n# \\"name\\": \\"deploy_k8s_cluster\\",\\n# \\"version\\": \\"1.0.0\\",\\n# \\"operations\\": 5\\n# },\\n# \\"warnings\\": [],\\n# \\"errors\\": []\\n# } # Strict mode (warnings cause failure)\\norch validate workflows/deploy.k --strict\\n# Error: Validation failed with warnings:\\n# - Operation \'create_servers\': Missing retry_policy\\n# - Operation \'install_k8s\': Resource limits not specified # Validate all workflows\\nls workflows/*.k | each { |file| let result = orch validate $file.name if $result.valid { echo $\\"✓ ($file.name)\\" } else { echo $\\"✗ ($file.name): ($result.errors | str join \', \')\\" }\\n} # CI/CD validation\\ntry { orch validate workflow.k --strict echo \\"✓ Validation passed\\"\\n} catch { echo \\"✗ Validation failed\\" exit 1\\n}\\n```plaintext **Validation Checks:** - ✅ KCL syntax correctness\\n- ✅ Required fields present (`name`, `version`, `operations`)\\n- ✅ Dependency graph valid (no cycles)\\n- ✅ Resource limits within bounds\\n- ✅ Provider configurations valid\\n- ✅ Operation types supported\\n- ⚠️ Optional: Retry policies defined\\n- ⚠️ Optional: Resource limits specified #### `orch tasks [--status ] [--limit ]` List orchestrator tasks from local state. **Flags:** - `--status `: Filter by status (`pending`, `running`, `completed`, `failed`)\\n- `--limit `: Limit results (default: 100)\\n- `--data-dir `: Data directory **Examples:** ```nushell\\n# All tasks (last 100)\\norch tasks\\n# [\\n# {\\n# \\"task_id\\": \\"task_abc123\\",\\n# \\"name\\": \\"deploy_kubernetes\\",\\n# \\"status\\": \\"running\\",\\n# \\"priority\\": 5,\\n# \\"created_at\\": \\"2025-10-09T12:00:00Z\\",\\n# \\"progress\\": 45\\n# }\\n# ] # Running tasks only\\norch tasks --status running # Failed tasks (last 10)\\norch tasks --status failed --limit 10 # Pending high-priority tasks\\norch tasks --status pending | where priority > 7 # Monitor active tasks\\nwatch { orch tasks --status running | select name progress updated_at | table\\n} # Count tasks by status\\norch tasks | group-by status | each { |group| { status: $group.0, count: ($group.1 | length) }\\n}\\n```plaintext ### Environment Variables | Variable | Description | Default |\\n|----------|-------------|---------|\\n| `ORCHESTRATOR_DATA_DIR` | Data directory | `provisioning/platform/orchestrator/data` | ### Performance Comparison | Operation | HTTP API | Plugin | Latency Reduction |\\n|-----------|----------|--------|-------------------|\\n| Status query | ~30ms | ~1ms | **97% faster** |\\n| Validate workflow | ~100ms | ~10ms | **90% faster** |\\n| List tasks | ~50ms | ~5ms | **90% faster** | **Use Case: CI/CD Pipeline** ```nushell\\n# HTTP approach (slow)\\nhttp get http://localhost:9090/tasks --status running | each { |task| http get $\\"http://localhost:9090/tasks/($task.id)\\" }\\n# Total: ~500ms for 10 tasks # Plugin approach (fast)\\norch tasks --status running\\n# Total: ~5ms for 10 tasks\\n# Result: 100x faster\\n```plaintext ### Troubleshooting Orchestrator **\\"Failed to read status\\"** ```bash\\n# Check data directory exists\\nls -la provisioning/platform/orchestrator/data/ # Create if missing\\nmkdir -p provisioning/platform/orchestrator/data # Check permissions (must be readable)\\nchmod 755 provisioning/platform/orchestrator/data\\n```plaintext **\\"Workflow validation failed\\"** ```nushell\\n# Use strict mode for detailed errors\\norch validate workflows/deploy.k --strict # Check KCL syntax manually\\nkcl fmt workflows/deploy.k\\nkcl run workflows/deploy.k\\n```plaintext **\\"No tasks found\\"** ```bash\\n# Check orchestrator running\\nps aux | grep orchestrator # Start orchestrator if not running\\ncd provisioning/platform/orchestrator\\n./scripts/start-orchestrator.nu --background # Check task files\\nls provisioning/platform/orchestrator/data/tasks/\\n```plaintext --- ## Integration Examples ### Example 1: Complete Authenticated Deployment Full workflow with authentication, secrets, and deployment: ```nushell\\n# Step 1: Login with MFA\\nauth login admin\\nauth mfa verify --code (input \\"MFA code: \\") # Step 2: Verify orchestrator health\\nif (orch status | get health) != \\"healthy\\" { error make { msg: \\"Orchestrator unhealthy\\" }\\n} # Step 3: Validate deployment workflow\\nlet validation = orch validate workflows/production-deploy.k --strict\\nif not $validation.valid { error make { msg: $\\"Validation failed: ($validation.errors)\\" }\\n} # Step 4: Encrypt production secrets\\nlet secrets = open secrets/production.yaml\\nkms encrypt ($secrets | to json) --backend rustyvault --key prod-main | save secrets/production.enc # Step 5: Submit deployment\\nprovisioning cluster create production --check # Step 6: Monitor progress\\nwhile (orch tasks --status running | length) > 0 { orch tasks --status running | select name progress updated_at | table sleep 10sec\\n} echo \\"✓ Deployment complete\\"\\n```plaintext ### Example 2: Batch Secret Rotation Rotate all secrets in multiple environments: ```nushell\\n# Rotate database passwords\\n[\\"dev\\", \\"staging\\", \\"production\\"] | each { |env| # Generate new password let new_password = (openssl rand -base64 32) # Encrypt with environment-specific key let encrypted = kms encrypt $new_password --backend rustyvault --key $\\"($env)-main\\" # Save encrypted password { environment: $env, password_enc: $encrypted, rotated_at: (date now | format date \\"%Y-%m-%d %H:%M:%S\\") } | save $\\"secrets/db-password-($env).json\\" echo $\\"✓ Rotated password for ($env)\\"\\n}\\n```plaintext ### Example 3: Multi-Environment Deployment Deploy to multiple environments with validation: ```nushell\\n# Define environments\\nlet environments = [ { name: \\"dev\\", validate: \\"basic\\" }, { name: \\"staging\\", validate: \\"strict\\" }, { name: \\"production\\", validate: \\"strict\\", mfa_required: true }\\n] # Deploy to each environment\\n$environments | each { |env| echo $\\"Deploying to ($env.name)...\\" # Authenticate if production if $env.mfa_required? { if not (auth verify | get mfa_verified) { auth mfa verify --code (input $\\"MFA code for ($env.name): \\") } } # Validate workflow let validation = if $env.validate == \\"strict\\" { orch validate $\\"workflows/($env.name)-deploy.k\\" --strict } else { orch validate $\\"workflows/($env.name)-deploy.k\\" } if not $validation.valid { echo $\\"✗ Validation failed for ($env.name)\\" continue } # Decrypt secrets let secrets = kms decrypt (open $\\"secrets/($env.name).enc\\") # Deploy provisioning cluster create $env.name echo $\\"✓ Deployed to ($env.name)\\"\\n}\\n```plaintext ### Example 4: Automated Backup and Encryption Backup configuration files with encryption: ```nushell\\n# Backup script\\nlet backup_dir = $\\"backups/(date now | format date \\"%Y%m%d-%H%M%S\\")\\"\\nmkdir $backup_dir # Backup and encrypt configs\\nls configs/**/*.yaml | each { |file| let encrypted = kms encrypt (open $file.name) --backend age let backup_path = $\\"($backup_dir)/($file.name | path basename).enc\\" $encrypted | save $backup_path echo $\\"✓ Backed up ($file.name)\\"\\n} # Create manifest\\n{ backup_date: (date now), files: (ls $\\"($backup_dir)/*.enc\\" | length), backend: \\"age\\"\\n} | save $\\"($backup_dir)/manifest.json\\" echo $\\"✓ Backup complete: ($backup_dir)\\"\\n```plaintext ### Example 5: Health Monitoring Dashboard Real-time health monitoring: ```nushell\\n# Health dashboard\\nwhile true { clear # Header echo \\"=== Provisioning Platform Health Dashboard ===\\" echo $\\"Updated: (date now | format date \\"%Y-%m-%d %H:%M:%S\\")\\" echo \\"\\" # Authentication status let auth_status = try { auth verify } catch { { active: false } } echo $\\"Auth: (if $auth_status.active { \'✓ Active\' } else { \'✗ Inactive\' })\\" # KMS status let kms_health = kms status echo $\\"KMS: (if $kms_health.status == \'healthy\' { \'✓ Healthy\' } else { \'✗ Unhealthy\' })\\" # Orchestrator status let orch_health = orch status echo $\\"Orchestrator: (if $orch_health.health == \'healthy\' { \'✓ Healthy\' } else { \'✗ Unhealthy\' })\\" echo $\\"Active Tasks: ($orch_health.active_tasks)\\" echo $\\"Failed Tasks: ($orch_health.failed_tasks)\\" # Task summary echo \\"\\" echo \\"=== Running Tasks ===\\" orch tasks --status running | select name progress updated_at | table sleep 10sec\\n}\\n```plaintext --- ## Best Practices ### When to Use Plugins vs HTTP **✅ Use Plugins When:** - Performance is critical (high-frequency operations)\\n- Working in pipelines (Nushell data structures)\\n- Need offline capability (KMS, orchestrator local ops)\\n- Building automation scripts\\n- CI/CD pipelines **Use HTTP When:** - Calling from external systems (not Nushell)\\n- Need consistent REST API interface\\n- Cross-language integration\\n- Web UI backend ### Performance Optimization **1. Batch Operations** ```nushell\\n# ❌ Slow: Individual HTTP calls in loop\\nls configs/*.yaml | each { |file| http post http://localhost:9998/encrypt { data: (open $file.name) }\\n}\\n# Total: ~5 seconds (50ms × 100) # ✅ Fast: Plugin in pipeline\\nls configs/*.yaml | each { |file| kms encrypt (open $file.name)\\n}\\n# Total: ~0.5 seconds (5ms × 100)\\n```plaintext **2. Parallel Processing** ```nushell\\n# Process multiple operations in parallel\\nls configs/*.yaml | par-each { |file| kms encrypt (open $file.name) | save $\\"encrypted/($file.name).enc\\" }\\n```plaintext **3. Caching Session State** ```nushell\\n# Cache auth verification\\nlet $auth_cache = auth verify\\nif $auth_cache.active { # Use cached result instead of repeated calls echo $\\"Authenticated as ($auth_cache.user)\\"\\n}\\n```plaintext ### Error Handling **Graceful Degradation:** ```nushell\\n# Try plugin, fallback to HTTP if unavailable\\ndef kms_encrypt [data: string] { try { kms encrypt $data } catch { http post http://localhost:9998/encrypt { data: $data } | get encrypted }\\n}\\n```plaintext **Comprehensive Error Handling:** ```nushell\\n# Handle all error cases\\ndef safe_deployment [] { # Check authentication let auth_status = try { auth verify } catch { echo \\"✗ Authentication failed, logging in...\\" auth login admin auth verify } # Check KMS health let kms_health = try { kms status } catch { error make { msg: \\"KMS unavailable, cannot proceed\\" } } # Validate workflow let validation = try { orch validate workflow.k --strict } catch { error make { msg: \\"Workflow validation failed\\" } } # Proceed if all checks pass if $auth_status.active and $kms_health.status == \\"healthy\\" and $validation.valid { echo \\"✓ All checks passed, deploying...\\" provisioning cluster create production }\\n}\\n```plaintext ### Security Best Practices **1. Never Log Decrypted Data** ```nushell\\n# ❌ BAD: Logs plaintext password\\nlet password = kms decrypt $encrypted_password\\necho $\\"Password: ($password)\\" # Visible in logs! # ✅ GOOD: Use directly without logging\\nlet password = kms decrypt $encrypted_password\\npsql --dbname mydb --password $password # Not logged\\n```plaintext **2. Use Context (AAD) for Critical Data** ```nushell\\n# Encrypt with context\\nlet context = $\\"user=(whoami),env=production,date=(date now | format date \\"%Y-%m-%d\\")\\"\\nkms encrypt $sensitive_data --context $context # Decrypt requires same context\\nkms decrypt $encrypted --context $context\\n```plaintext **3. Rotate Backup Codes** ```nushell\\n# After using backup code, generate new set\\nauth mfa verify --code ABCD-EFGH-IJKL\\n# Warning: Backup code used\\nauth mfa regenerate-backups\\n# New backup codes generated\\n```plaintext **4. Limit Token Lifetime** ```nushell\\n# Check token expiration before long operations\\nlet session = auth verify\\nlet expires_in = (($session.expires_at | into datetime) - (date now))\\nif $expires_in < 5min { echo \\"⚠️ Token expiring soon, re-authenticating...\\" auth login $session.user\\n}\\n```plaintext --- ## Troubleshooting ### Common Issues Across Plugins **\\"Plugin not found\\"** ```bash\\n# Check plugin registration\\nplugin list | where name =~ \\"auth|kms|orch\\" # Re-register if missing\\ncd provisioning/core/plugins/nushell-plugins\\nplugin add target/release/nu_plugin_auth\\nplugin add target/release/nu_plugin_kms\\nplugin add target/release/nu_plugin_orchestrator # Restart Nushell\\nexit\\nnu\\n```plaintext **\\"Plugin command failed\\"** ```nushell\\n# Enable debug mode\\n$env.RUST_LOG = \\"debug\\" # Run command again to see detailed errors\\nkms encrypt \\"test\\" # Check plugin version compatibility\\nplugin list | where name =~ \\"kms\\" | select name version\\n```plaintext **\\"Permission denied\\"** ```bash\\n# Check plugin executable permissions\\nls -l provisioning/core/plugins/nushell-plugins/target/release/nu_plugin_*\\n# Should show: -rwxr-xr-x # Fix if needed\\nchmod +x provisioning/core/plugins/nushell-plugins/target/release/nu_plugin_*\\n```plaintext ### Platform-Specific Issues **macOS Issues:** ```bash\\n# \\"cannot be opened because the developer cannot be verified\\"\\nxattr -d com.apple.quarantine target/release/nu_plugin_auth\\nxattr -d com.apple.quarantine target/release/nu_plugin_kms\\nxattr -d com.apple.quarantine target/release/nu_plugin_orchestrator # Keychain access denied\\n# System Preferences → Security & Privacy → Privacy → Full Disk Access\\n# Add: /usr/local/bin/nu\\n```plaintext **Linux Issues:** ```bash\\n# Keyring service not running\\nsystemctl --user status gnome-keyring-daemon\\nsystemctl --user start gnome-keyring-daemon # Missing dependencies\\nsudo apt install libssl-dev pkg-config # Ubuntu/Debian\\nsudo dnf install openssl-devel # Fedora\\n```plaintext **Windows Issues:** ```powershell\\n# Credential Manager access denied\\n# Control Panel → User Accounts → Credential Manager\\n# Ensure Windows Credential Manager service is running # Missing Visual C++ runtime\\n# Download from: https://aka.ms/vs/17/release/vc_redist.x64.exe\\n```plaintext ### Debugging Techniques **Enable Verbose Logging:** ```nushell\\n# Set log level\\n$env.RUST_LOG = \\"debug,nu_plugin_auth=trace\\" # Run command\\nauth login admin # Check logs\\n```plaintext **Test Plugin Directly:** ```bash\\n# Test plugin communication (advanced)\\necho \'{\\"Call\\": [0, {\\"name\\": \\"auth\\", \\"call\\": \\"login\\", \\"args\\": [\\"admin\\", \\"password\\"]}]}\' \\\\ | target/release/nu_plugin_auth\\n```plaintext **Check Plugin Health:** ```nushell\\n# Test each plugin\\nauth --help # Should show auth commands\\nkms --help # Should show kms commands\\norch --help # Should show orch commands # Test functionality\\nauth verify # Should return session status\\nkms status # Should return backend status\\norch status # Should return orchestrator status\\n```plaintext --- ## Migration Guide ### Migrating from HTTP to Plugin-Based **Phase 1: Install Plugins (No Breaking Changes)** ```bash\\n# Build and register plugins\\ncd provisioning/core/plugins/nushell-plugins\\ncargo build --release --all\\nplugin add target/release/nu_plugin_auth\\nplugin add target/release/nu_plugin_kms\\nplugin add target/release/nu_plugin_orchestrator # Verify HTTP still works\\nhttp get http://localhost:9090/health\\n```plaintext **Phase 2: Update Scripts Incrementally** ```nushell\\n# Before (HTTP)\\ndef encrypt_config [file: string] { let data = open $file let result = http post http://localhost:9998/encrypt { data: $data } $result.encrypted | save $\\"($file).enc\\"\\n} # After (Plugin with fallback)\\ndef encrypt_config [file: string] { let data = open $file let encrypted = try { kms encrypt $data --backend rustyvault } catch { # Fallback to HTTP if plugin unavailable (http post http://localhost:9998/encrypt { data: $data }).encrypted } $encrypted | save $\\"($file).enc\\"\\n}\\n```plaintext **Phase 3: Test Migration** ```nushell\\n# Run side-by-side comparison\\ndef test_migration [] { let test_data = \\"test secret data\\" # Plugin approach let start_plugin = date now let plugin_result = kms encrypt $test_data let plugin_time = ((date now) - $start_plugin) # HTTP approach let start_http = date now let http_result = (http post http://localhost:9998/encrypt { data: $test_data }).encrypted let http_time = ((date now) - $start_http) echo $\\"Plugin: ($plugin_time)ms\\" echo $\\"HTTP: ($http_time)ms\\" echo $\\"Speedup: (($http_time / $plugin_time))x\\"\\n}\\n```plaintext **Phase 4: Gradual Rollout** ```nushell\\n# Use feature flag for controlled rollout\\n$env.USE_PLUGINS = true def encrypt_with_flag [data: string] { if $env.USE_PLUGINS { kms encrypt $data } else { (http post http://localhost:9998/encrypt { data: $data }).encrypted }\\n}\\n```plaintext **Phase 5: Full Migration** ```nushell\\n# Replace all HTTP calls with plugin calls\\n# Remove fallback logic once stable\\ndef encrypt_config [file: string] { let data = open $file kms encrypt $data --backend rustyvault | save $\\"($file).enc\\"\\n}\\n```plaintext ### Rollback Strategy ```nushell\\n# If issues arise, quickly rollback\\ndef rollback_to_http [] { # Remove plugin registrations plugin rm nu_plugin_auth plugin rm nu_plugin_kms plugin rm nu_plugin_orchestrator # Restart Nushell exec nu\\n}\\n```plaintext --- ## Advanced Configuration ### Custom Plugin Paths ```nushell\\n# ~/.config/nushell/config.nu\\n$env.PLUGIN_PATH = \\"/opt/provisioning/plugins\\" # Register from custom location\\nplugin add $\\"($env.PLUGIN_PATH)/nu_plugin_auth\\"\\nplugin add $\\"($env.PLUGIN_PATH)/nu_plugin_kms\\"\\nplugin add $\\"($env.PLUGIN_PATH)/nu_plugin_orchestrator\\"\\n```plaintext ### Environment-Specific Configuration ```nushell\\n# ~/.config/nushell/env.nu # Development environment\\nif ($env.ENV? == \\"dev\\") { $env.RUSTYVAULT_ADDR = \\"http://localhost:8200\\" $env.CONTROL_CENTER_URL = \\"http://localhost:3000\\"\\n} # Staging environment\\nif ($env.ENV? == \\"staging\\") { $env.RUSTYVAULT_ADDR = \\"https://vault-staging.example.com\\" $env.CONTROL_CENTER_URL = \\"https://control-staging.example.com\\"\\n} # Production environment\\nif ($env.ENV? == \\"prod\\") { $env.RUSTYVAULT_ADDR = \\"https://vault.example.com\\" $env.CONTROL_CENTER_URL = \\"https://control.example.com\\"\\n}\\n```plaintext ### Plugin Aliases ```nushell\\n# ~/.config/nushell/config.nu # Auth shortcuts\\nalias login = auth login\\nalias logout = auth logout\\nalias whoami = auth verify | get user # KMS shortcuts\\nalias encrypt = kms encrypt\\nalias decrypt = kms decrypt # Orchestrator shortcuts\\nalias status = orch status\\nalias tasks = orch tasks\\nalias validate = orch validate\\n```plaintext ### Custom Commands ```nushell\\n# ~/.config/nushell/custom_commands.nu # Encrypt all files in directory\\ndef encrypt-dir [dir: string] { ls $\\"($dir)/**/*\\" | where type == file | each { |file| kms encrypt (open $file.name) | save $\\"($file.name).enc\\" echo $\\"✓ Encrypted ($file.name)\\" }\\n} # Decrypt all files in directory\\ndef decrypt-dir [dir: string] { ls $\\"($dir)/**/*.enc\\" | each { |file| kms decrypt (open $file.name) | save (echo $file.name | str replace \'.enc\' \'\') echo $\\"✓ Decrypted ($file.name)\\" }\\n} # Monitor deployments\\ndef watch-deployments [] { while true { clear echo \\"=== Active Deployments ===\\" orch tasks --status running | table sleep 5sec }\\n}\\n```plaintext --- ## Security Considerations ### Threat Model **What Plugins Protect Against:** - ✅ Network eavesdropping (no HTTP for KMS/orch)\\n- ✅ Token theft from files (keyring storage)\\n- ✅ Credential exposure in logs (prompt-based input)\\n- ✅ Man-in-the-middle attacks (local file access) **What Plugins Don\'t Protect Against:** - ❌ Memory dumping (decrypted data in RAM)\\n- ❌ Malicious plugins (trust registry only)\\n- ❌ Compromised OS keyring\\n- ❌ Physical access to machine ### Secure Deployment **1. Verify Plugin Integrity** ```bash\\n# Check plugin signatures (if available)\\nsha256sum target/release/nu_plugin_auth\\n# Compare with published checksums # Build from trusted source\\ngit clone https://github.com/provisioning-platform/plugins\\ncd plugins\\ncargo build --release --all\\n```plaintext **2. Restrict Plugin Access** ```bash\\n# Set plugin permissions (only owner can execute)\\nchmod 700 target/release/nu_plugin_* # Store in protected directory\\nsudo mkdir -p /opt/provisioning/plugins\\nsudo chown $(whoami):$(whoami) /opt/provisioning/plugins\\nsudo chmod 755 /opt/provisioning/plugins\\nmv target/release/nu_plugin_* /opt/provisioning/plugins/\\n```plaintext **3. Audit Plugin Usage** ```nushell\\n# Log plugin calls (for compliance)\\ndef logged_encrypt [data: string] { let timestamp = date now let result = kms encrypt $data { timestamp: $timestamp, action: \\"encrypt\\" } | save --append audit.log $result\\n}\\n```plaintext **4. Rotate Credentials Regularly** ```nushell\\n# Weekly credential rotation script\\ndef rotate_credentials [] { # Re-authenticate auth logout auth login admin # Rotate KMS keys (if supported) kms rotate-key --key provisioning-main # Update encrypted secrets ls secrets/*.enc | each { |file| let plain = kms decrypt (open $file.name) kms encrypt $plain | save $file.name }\\n}\\n```plaintext --- ## FAQ **Q: Can I use plugins without RustyVault/Age installed?** A: Yes, authentication and orchestrator plugins work independently. KMS plugin requires at least one backend configured (Age is easiest for local dev). **Q: Do plugins work in CI/CD pipelines?** A: Yes, plugins work great in CI/CD. For headless environments (no keyring), use environment variables for auth or file-based tokens. ```bash\\n# CI/CD example\\nexport CONTROL_CENTER_TOKEN=\\"jwt-token-here\\"\\nkms encrypt \\"data\\" --backend age\\n```plaintext **Q: How do I update plugins?** A: Rebuild and re-register: ```bash\\ncd provisioning/core/plugins/nushell-plugins\\ngit pull\\ncargo build --release --all\\nplugin add --force target/release/nu_plugin_auth\\nplugin add --force target/release/nu_plugin_kms\\nplugin add --force target/release/nu_plugin_orchestrator\\n```plaintext **Q: Can I use multiple KMS backends simultaneously?** A: Yes, specify `--backend` for each operation: ```nushell\\nkms encrypt \\"data1\\" --backend rustyvault\\nkms encrypt \\"data2\\" --backend age\\nkms encrypt \\"data3\\" --backend aws\\n```plaintext **Q: What happens if a plugin crashes?** A: Nushell isolates plugin crashes. The command fails with an error, but Nushell continues running. Check logs with `$env.RUST_LOG = \\"debug\\"`. **Q: Are plugins compatible with older Nushell versions?** A: Plugins require Nushell 0.107.1+. For older versions, use HTTP API. **Q: How do I backup MFA enrollment?** A: Save backup codes securely (password manager, encrypted file). QR code can be re-scanned from the same secret. ```nushell\\n# Save backup codes\\nauth mfa enroll totp | save mfa-backup-codes.txt\\nkms encrypt (open mfa-backup-codes.txt) | save mfa-backup-codes.enc\\nrm mfa-backup-codes.txt\\n```plaintext **Q: Can plugins work offline?** A: Partially: - ✅ `kms` with Age backend (fully offline)\\n- ✅ `orch` status/tasks (reads local files)\\n- ❌ `auth` (requires control center)\\n- ❌ `kms` with RustyVault/AWS/Vault (requires network) **Q: How do I troubleshoot plugin performance?** A: Use Nushell\'s timing: ```nushell\\ntimeit { kms encrypt \\"data\\" }\\n# 5ms 123μs 456ns timeit { http post http://localhost:9998/encrypt { data: \\"data\\" } }\\n# 52ms 789μs 123ns\\n```plaintext --- ## Related Documentation - **Security System**: `/Users/Akasha/project-provisioning/docs/architecture/ADR-009-security-system-complete.md`\\n- **JWT Authentication**: `/Users/Akasha/project-provisioning/docs/architecture/JWT_AUTH_IMPLEMENTATION.md`\\n- **Config Encryption**: `/Users/Akasha/project-provisioning/docs/user/CONFIG_ENCRYPTION_GUIDE.md`\\n- **RustyVault Integration**: `/Users/Akasha/project-provisioning/RUSTYVAULT_INTEGRATION_SUMMARY.md`\\n- **MFA Implementation**: `/Users/Akasha/project-provisioning/docs/architecture/MFA_IMPLEMENTATION_SUMMARY.md`\\n- **Nushell Plugins Reference**: `/Users/Akasha/project-provisioning/docs/user/NUSHELL_PLUGINS_GUIDE.md` --- **Version**: 1.0.0\\n**Maintained By**: Platform Team\\n**Last Updated**: 2025-10-09\\n**Feedback**: Open an issue or contact ","breadcrumbs":"Plugin Integration Guide » Architecture Benefits","id":"1617","title":"Architecture Benefits"},"1618":{"body":"Complete guide to authentication, KMS, and orchestrator plugins.","breadcrumbs":"NuShell Plugins Guide » Nushell Plugins for Provisioning Platform","id":"1618","title":"Nushell Plugins for Provisioning Platform"},"1619":{"body":"Three native Nushell plugins provide high-performance integration with the provisioning platform: nu_plugin_auth - JWT authentication and MFA operations nu_plugin_kms - Key management (RustyVault, Age, Cosmian, AWS, Vault) nu_plugin_orchestrator - Orchestrator operations (status, validate, tasks)","breadcrumbs":"NuShell Plugins Guide » Overview","id":"1619","title":"Overview"},"162":{"body":"Edit the generated configuration: # Edit with your preferred editor\\n$EDITOR workspace/infra/my-infra/settings.k Example configuration: import provisioning.settings as cfg # Infrastructure settings\\ninfra_settings = cfg.InfraSettings { name = \\"my-infra\\" provider = \\"local\\" # Start with local provider environment = \\"development\\"\\n} # Server configuration\\nservers = [ { hostname = \\"dev-server-01\\" cores = 2 memory = 4096 # MB disk = 50 # GB }\\n]","breadcrumbs":"First Deployment » Step 2: Edit Configuration","id":"162","title":"Step 2: Edit Configuration"},"1620":{"body":"Performance Advantages : 10x faster than HTTP API calls (KMS operations) Direct access to Rust libraries (no HTTP overhead) Native integration with Nushell pipelines Type safety with Nushell\'s type system Developer Experience : Pipeline friendly - Use Nushell pipes naturally Tab completion - All commands and flags Consistent interface - Follows Nushell conventions Error handling - Nushell-native error messages","breadcrumbs":"NuShell Plugins Guide » Why Native Plugins?","id":"1620","title":"Why Native Plugins?"},"1621":{"body":"","breadcrumbs":"NuShell Plugins Guide » Installation","id":"1621","title":"Installation"},"1622":{"body":"Nushell 0.107.1+ Rust toolchain (for building from source) Access to provisioning platform services","breadcrumbs":"NuShell Plugins Guide » Prerequisites","id":"1622","title":"Prerequisites"},"1623":{"body":"cd /Users/Akasha/project-provisioning/provisioning/core/plugins/nushell-plugins # Build all plugins\\ncargo build --release -p nu_plugin_auth\\ncargo build --release -p nu_plugin_kms\\ncargo build --release -p nu_plugin_orchestrator # Or build individually\\ncargo build --release -p nu_plugin_auth\\ncargo build --release -p nu_plugin_kms\\ncargo build --release -p nu_plugin_orchestrator\\n```plaintext ### Register with Nushell ```bash\\n# Register all plugins\\nplugin add target/release/nu_plugin_auth\\nplugin add target/release/nu_plugin_kms\\nplugin add target/release/nu_plugin_orchestrator # Verify registration\\nplugin list | where name =~ \\"provisioning\\"\\n```plaintext ### Verify Installation ```bash\\n# Test auth commands\\nauth --help # Test KMS commands\\nkms --help # Test orchestrator commands\\norch --help\\n```plaintext --- ## Plugin: nu_plugin_auth Authentication plugin for JWT login, MFA enrollment, and session management. ### Commands #### `auth login  [password]` Login to provisioning platform and store JWT tokens securely. **Arguments**: - `username` (required): Username for authentication\\n- `password` (optional): Password (prompts interactively if not provided) **Flags**: - `--url `: Control center URL (default: `http://localhost:9080`)\\n- `--password `: Password (alternative to positional argument) **Examples**: ```nushell\\n# Interactive password prompt (recommended)\\nauth login admin # Password in command (not recommended for production)\\nauth login admin mypassword # Custom URL\\nauth login admin --url http://control-center:9080 # Pipeline usage\\n\\"admin\\" | auth login\\n```plaintext **Token Storage**:\\nTokens are stored securely in OS-native keyring: - **macOS**: Keychain Access\\n- **Linux**: Secret Service (gnome-keyring, kwallet)\\n- **Windows**: Credential Manager **Success Output**: ```plaintext\\n✓ Login successful\\nUser: admin\\nRole: Admin\\nExpires: 2025-10-09T14:30:00Z\\n```plaintext --- #### `auth logout` Logout from current session and remove stored tokens. **Examples**: ```nushell\\n# Simple logout\\nauth logout # Pipeline usage (conditional logout)\\nif (auth verify | get active) { auth logout }\\n```plaintext **Success Output**: ```plaintext\\n✓ Logged out successfully\\n```plaintext --- #### `auth verify` Verify current session and check token validity. **Examples**: ```nushell\\n# Check session status\\nauth verify # Pipeline usage\\nauth verify | if $in.active { echo \\"Session valid\\" } else { echo \\"Session expired\\" }\\n```plaintext **Success Output**: ```json\\n{ \\"active\\": true, \\"user\\": \\"admin\\", \\"role\\": \\"Admin\\", \\"expires_at\\": \\"2025-10-09T14:30:00Z\\", \\"mfa_verified\\": true\\n}\\n```plaintext --- #### `auth sessions` List all active sessions for current user. **Examples**: ```nushell\\n# List sessions\\nauth sessions # Filter by date\\nauth sessions | where created_at > (date now | date to-timezone UTC | into string)\\n```plaintext **Output Format**: ```json\\n[ { \\"session_id\\": \\"sess_abc123\\", \\"created_at\\": \\"2025-10-09T12:00:00Z\\", \\"expires_at\\": \\"2025-10-09T14:30:00Z\\", \\"ip_address\\": \\"192.168.1.100\\", \\"user_agent\\": \\"nushell/0.107.1\\" }\\n]\\n```plaintext --- #### `auth mfa enroll ` Enroll in MFA (TOTP or WebAuthn). **Arguments**: - `type` (required): MFA type (`totp` or `webauthn`) **Examples**: ```nushell\\n# Enroll TOTP (Google Authenticator, Authy)\\nauth mfa enroll totp # Enroll WebAuthn (YubiKey, Touch ID, Windows Hello)\\nauth mfa enroll webauthn\\n```plaintext **TOTP Enrollment Output**: ```plaintext\\n✓ TOTP enrollment initiated Scan this QR code with your authenticator app: ████ ▄▄▄▄▄ █▀█ █▄▀▀▀▄ ▄▄▄▄▄ ████ ████ █ █ █▀▀▀█▄ ▀▀█ █ █ ████ ████ █▄▄▄█ █ █▀▄ ▀▄▄█ █▄▄▄█ ████ ... Or enter manually:\\nSecret: JBSWY3DPEHPK3PXP\\nURL: otpauth://totp/Provisioning:admin?secret=JBSWY3DPEHPK3PXP&issuer=Provisioning Backup codes (save securely):\\n1. ABCD-EFGH-IJKL\\n2. MNOP-QRST-UVWX\\n...\\n```plaintext --- #### `auth mfa verify --code ` Verify MFA code (TOTP or backup code). **Flags**: - `--code ` (required): 6-digit TOTP code or backup code **Examples**: ```nushell\\n# Verify TOTP code\\nauth mfa verify --code 123456 # Verify backup code\\nauth mfa verify --code ABCD-EFGH-IJKL\\n```plaintext **Success Output**: ```plaintext\\n✓ MFA verification successful\\n```plaintext --- ### Environment Variables | Variable | Description | Default |\\n|----------|-------------|---------|\\n| `USER` | Default username | Current OS user |\\n| `CONTROL_CENTER_URL` | Control center URL | `http://localhost:9080` | --- ### Error Handling **Common Errors**: ```nushell\\n# \\"No active session\\"\\nError: No active session found\\n→ Run: auth login  # \\"Invalid credentials\\"\\nError: Authentication failed: Invalid username or password\\n→ Check username and password # \\"Token expired\\"\\nError: Token has expired\\n→ Run: auth login  # \\"MFA required\\"\\nError: MFA verification required\\n→ Run: auth mfa verify --code  # \\"Keyring error\\" (macOS)\\nError: Failed to access keyring\\n→ Check Keychain Access permissions # \\"Keyring error\\" (Linux)\\nError: Failed to access keyring\\n→ Install gnome-keyring or kwallet\\n```plaintext --- ## Plugin: nu_plugin_kms Key Management Service plugin supporting multiple backends. ### Supported Backends | Backend | Description | Use Case |\\n|---------|-------------|----------|\\n| `rustyvault` | RustyVault Transit engine | Production KMS |\\n| `age` | Age encryption (local) | Development/testing |\\n| `cosmian` | Cosmian KMS (HTTP) | Cloud KMS |\\n| `aws` | AWS KMS | AWS environments |\\n| `vault` | HashiCorp Vault | Enterprise KMS | ### Commands #### `kms encrypt  [--backend ]` Encrypt data using KMS. **Arguments**: - `data` (required): Data to encrypt (string or binary) **Flags**: - `--backend `: KMS backend (`rustyvault`, `age`, `cosmian`, `aws`, `vault`)\\n- `--key `: Key ID or recipient (backend-specific)\\n- `--context `: Additional authenticated data (AAD) **Examples**: ```nushell\\n# Auto-detect backend from environment\\nkms encrypt \\"secret data\\" # RustyVault\\nkms encrypt \\"data\\" --backend rustyvault --key provisioning-main # Age (local encryption)\\nkms encrypt \\"data\\" --backend age --key age1xxxxxxxxx # AWS KMS\\nkms encrypt \\"data\\" --backend aws --key alias/provisioning # With context (AAD)\\nkms encrypt \\"data\\" --backend rustyvault --key provisioning-main --context \\"user=admin\\"\\n```plaintext **Output Format**: ```plaintext\\nvault:v1:abc123def456...\\n```plaintext --- #### `kms decrypt  [--backend ]` Decrypt KMS-encrypted data. **Arguments**: - `encrypted` (required): Encrypted data (base64 or KMS format) **Flags**: - `--backend `: KMS backend (auto-detected if not specified)\\n- `--context `: Additional authenticated data (AAD, must match encryption) **Examples**: ```nushell\\n# Auto-detect backend\\nkms decrypt \\"vault:v1:abc123def456...\\" # RustyVault explicit\\nkms decrypt \\"vault:v1:abc123...\\" --backend rustyvault # Age\\nkms decrypt \\"-----BEGIN AGE ENCRYPTED FILE-----...\\" --backend age # With context\\nkms decrypt \\"vault:v1:abc123...\\" --backend rustyvault --context \\"user=admin\\"\\n```plaintext **Output**: ```plaintext\\nsecret data\\n```plaintext --- #### `kms generate-key [--spec ]` Generate data encryption key (DEK) using KMS. **Flags**: - `--spec `: Key specification (`AES128` or `AES256`, default: `AES256`)\\n- `--backend `: KMS backend **Examples**: ```nushell\\n# Generate AES-256 key\\nkms generate-key # Generate AES-128 key\\nkms generate-key --spec AES128 # Specific backend\\nkms generate-key --backend rustyvault\\n```plaintext **Output Format**: ```json\\n{ \\"plaintext\\": \\"base64-encoded-key\\", \\"ciphertext\\": \\"vault:v1:encrypted-key\\", \\"spec\\": \\"AES256\\"\\n}\\n```plaintext --- #### `kms status` Show KMS backend status and configuration. **Examples**: ```nushell\\n# Show status\\nkms status # Filter to specific backend\\nkms status | where backend == \\"rustyvault\\"\\n```plaintext **Output Format**: ```json\\n{ \\"backend\\": \\"rustyvault\\", \\"status\\": \\"healthy\\", \\"url\\": \\"http://localhost:8200\\", \\"mount_point\\": \\"transit\\", \\"version\\": \\"0.1.0\\"\\n}\\n```plaintext --- ### Environment Variables **RustyVault Backend**: ```bash\\nexport RUSTYVAULT_ADDR=\\"http://localhost:8200\\"\\nexport RUSTYVAULT_TOKEN=\\"your-token-here\\"\\nexport RUSTYVAULT_MOUNT=\\"transit\\"\\n```plaintext **Age Backend**: ```bash\\nexport AGE_RECIPIENT=\\"age1xxxxxxxxx\\"\\nexport AGE_IDENTITY=\\"/path/to/key.txt\\"\\n```plaintext **HTTP Backend (Cosmian)**: ```bash\\nexport KMS_HTTP_URL=\\"http://localhost:9998\\"\\nexport KMS_HTTP_BACKEND=\\"cosmian\\"\\n```plaintext **AWS KMS**: ```bash\\nexport AWS_REGION=\\"us-east-1\\"\\nexport AWS_ACCESS_KEY_ID=\\"...\\"\\nexport AWS_SECRET_ACCESS_KEY=\\"...\\"\\n```plaintext --- ### Performance Comparison | Operation | HTTP API | Plugin | Improvement |\\n|-----------|----------|--------|-------------|\\n| Encrypt (RustyVault) | ~50ms | ~5ms | **10x faster** |\\n| Decrypt (RustyVault) | ~50ms | ~5ms | **10x faster** |\\n| Encrypt (Age) | ~30ms | ~3ms | **10x faster** |\\n| Decrypt (Age) | ~30ms | ~3ms | **10x faster** |\\n| Generate Key | ~60ms | ~8ms | **7.5x faster** | --- ## Plugin: nu_plugin_orchestrator Orchestrator operations plugin for status, validation, and task management. ### Commands #### `orch status [--data-dir ]` Get orchestrator status from local files (no HTTP). **Flags**: - `--data-dir `: Data directory (default: `provisioning/platform/orchestrator/data`) **Examples**: ```nushell\\n# Default data dir\\norch status # Custom dir\\norch status --data-dir ./custom/data # Pipeline usage\\norch status | if $in.active_tasks > 0 { echo \\"Tasks running\\" }\\n```plaintext **Output Format**: ```json\\n{ \\"active_tasks\\": 5, \\"completed_tasks\\": 120, \\"failed_tasks\\": 2, \\"pending_tasks\\": 3, \\"uptime\\": \\"2d 4h 15m\\", \\"health\\": \\"healthy\\"\\n}\\n```plaintext --- #### `orch validate  [--strict]` Validate workflow KCL file. **Arguments**: - `workflow.k` (required): Path to KCL workflow file **Flags**: - `--strict`: Enable strict validation (all checks, warnings as errors) **Examples**: ```nushell\\n# Basic validation\\norch validate workflows/deploy.k # Strict mode\\norch validate workflows/deploy.k --strict # Pipeline usage\\nls workflows/*.k | each { |file| orch validate $file.name }\\n```plaintext **Output Format**: ```json\\n{ \\"valid\\": true, \\"workflow\\": { \\"name\\": \\"deploy_k8s_cluster\\", \\"version\\": \\"1.0.0\\", \\"operations\\": 5 }, \\"warnings\\": [], \\"errors\\": []\\n}\\n```plaintext **Validation Checks**: - KCL syntax errors\\n- Required fields present\\n- Dependency graph valid (no cycles)\\n- Resource limits within bounds\\n- Provider configurations valid --- #### `orch tasks [--status ] [--limit ]` List orchestrator tasks. **Flags**: - `--status `: Filter by status (`pending`, `running`, `completed`, `failed`)\\n- `--limit `: Limit number of results (default: 100)\\n- `--data-dir `: Data directory (default from `ORCHESTRATOR_DATA_DIR`) **Examples**: ```nushell\\n# All tasks\\norch tasks # Pending tasks only\\norch tasks --status pending # Running tasks (limit to 10)\\norch tasks --status running --limit 10 # Pipeline usage\\norch tasks --status failed | each { |task| echo $\\"Failed: ($task.name)\\" }\\n```plaintext **Output Format**: ```json\\n[ { \\"task_id\\": \\"task_abc123\\", \\"name\\": \\"deploy_kubernetes\\", \\"status\\": \\"running\\", \\"priority\\": 5, \\"created_at\\": \\"2025-10-09T12:00:00Z\\", \\"updated_at\\": \\"2025-10-09T12:05:00Z\\", \\"progress\\": 45 }\\n]\\n```plaintext --- ### Environment Variables | Variable | Description | Default |\\n|----------|-------------|---------|\\n| `ORCHESTRATOR_DATA_DIR` | Data directory | `provisioning/platform/orchestrator/data` | --- ### Performance Comparison | Operation | HTTP API | Plugin | Improvement |\\n|-----------|----------|--------|-------------|\\n| Status | ~30ms | ~3ms | **10x faster** |\\n| Validate | ~100ms | ~10ms | **10x faster** |\\n| Tasks List | ~50ms | ~5ms | **10x faster** | --- ## Pipeline Examples ### Authentication Flow ```nushell\\n# Login and verify in one pipeline\\nauth login admin | if $in.success { auth verify } | if $in.mfa_required { auth mfa verify --code (input \\"MFA code: \\") }\\n```plaintext ### KMS Operations ```nushell\\n# Encrypt multiple secrets\\n[\\"secret1\\", \\"secret2\\", \\"secret3\\"] | each { |data| kms encrypt $data --backend rustyvault } | save encrypted_secrets.json # Decrypt and process\\nopen encrypted_secrets.json | each { |enc| kms decrypt $enc } | each { |plain| echo $\\"Decrypted: ($plain)\\" }\\n```plaintext ### Orchestrator Monitoring ```nushell\\n# Monitor running tasks\\nwhile true { orch tasks --status running | each { |task| echo $\\"($task.name): ($task.progress)%\\" } sleep 5sec\\n}\\n```plaintext ### Combined Workflow ```nushell\\n# Complete deployment workflow\\nauth login admin | auth mfa verify --code (input \\"MFA: \\") | orch validate workflows/deploy.k | if $in.valid { orch tasks --status pending | where priority > 5 | each { |task| echo $\\"High priority: ($task.name)\\" } }\\n```plaintext --- ## Troubleshooting ### Auth Plugin **\\"No active session\\"**: ```nushell\\nauth login \\n```plaintext **\\"Keyring error\\" (macOS)**: - Check Keychain Access permissions\\n- Security & Privacy → Privacy → Full Disk Access → Add Nushell **\\"Keyring error\\" (Linux)**: ```bash\\n# Install keyring service\\nsudo apt install gnome-keyring # Ubuntu/Debian\\nsudo dnf install gnome-keyring # Fedora # Or use KWallet\\nsudo apt install kwalletmanager\\n```plaintext **\\"MFA verification failed\\"**: - Check time synchronization (TOTP requires accurate clocks)\\n- Use backup codes if TOTP not working\\n- Re-enroll MFA if device lost --- ### KMS Plugin **\\"RustyVault connection failed\\"**: ```bash\\n# Check RustyVault running\\ncurl http://localhost:8200/v1/sys/health # Set environment\\nexport RUSTYVAULT_ADDR=\\"http://localhost:8200\\"\\nexport RUSTYVAULT_TOKEN=\\"your-token\\"\\n```plaintext **\\"Age encryption failed\\"**: ```bash\\n# Check Age keys\\nls -la ~/.age/ # Generate new key if needed\\nage-keygen -o ~/.age/key.txt # Set environment\\nexport AGE_RECIPIENT=\\"age1xxxxxxxxx\\"\\nexport AGE_IDENTITY=\\"$HOME/.age/key.txt\\"\\n```plaintext **\\"AWS KMS access denied\\"**: ```bash\\n# Check AWS credentials\\naws sts get-caller-identity # Check KMS key policy\\naws kms describe-key --key-id alias/provisioning\\n```plaintext --- ### Orchestrator Plugin **\\"Failed to read status\\"**: ```bash\\n# Check data directory exists\\nls provisioning/platform/orchestrator/data/ # Create if missing\\nmkdir -p provisioning/platform/orchestrator/data\\n```plaintext **\\"Workflow validation failed\\"**: ```nushell\\n# Use strict mode for detailed errors\\norch validate workflows/deploy.k --strict\\n```plaintext **\\"No tasks found\\"**: ```bash\\n# Check orchestrator running\\nps aux | grep orchestrator # Start orchestrator\\ncd provisioning/platform/orchestrator\\n./scripts/start-orchestrator.nu --background\\n```plaintext --- ## Development ### Building from Source ```bash\\ncd provisioning/core/plugins/nushell-plugins # Clean build\\ncargo clean # Build with debug info\\ncargo build -p nu_plugin_auth\\ncargo build -p nu_plugin_kms\\ncargo build -p nu_plugin_orchestrator # Run tests\\ncargo test -p nu_plugin_auth\\ncargo test -p nu_plugin_kms\\ncargo test -p nu_plugin_orchestrator # Run all tests\\ncargo test --all\\n```plaintext ### Adding to CI/CD ```yaml\\nname: Build Nushell Plugins on: [push, pull_request] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Install Rust uses: actions-rs/toolchain@v1 with: toolchain: stable - name: Build Plugins run: | cd provisioning/core/plugins/nushell-plugins cargo build --release --all - name: Test Plugins run: | cd provisioning/core/plugins/nushell-plugins cargo test --all - name: Upload Artifacts uses: actions/upload-artifact@v3 with: name: plugins path: provisioning/core/plugins/nushell-plugins/target/release/nu_plugin_*\\n```plaintext --- ## Advanced Usage ### Custom Plugin Configuration Create `~/.config/nushell/plugin_config.nu`: ```nushell\\n# Auth plugin defaults\\n$env.CONTROL_CENTER_URL = \\"https://control-center.example.com\\" # KMS plugin defaults\\n$env.RUSTYVAULT_ADDR = \\"https://vault.example.com:8200\\"\\n$env.RUSTYVAULT_MOUNT = \\"transit\\" # Orchestrator plugin defaults\\n$env.ORCHESTRATOR_DATA_DIR = \\"/opt/orchestrator/data\\"\\n```plaintext ### Plugin Aliases Add to `~/.config/nushell/config.nu`: ```nushell\\n# Auth shortcuts\\nalias login = auth login\\nalias logout = auth logout # KMS shortcuts\\nalias encrypt = kms encrypt\\nalias decrypt = kms decrypt # Orchestrator shortcuts\\nalias status = orch status\\nalias validate = orch validate\\nalias tasks = orch tasks\\n```plaintext --- ## Security Best Practices ### Authentication ✅ **DO**: Use interactive password prompts\\n✅ **DO**: Enable MFA for production environments\\n✅ **DO**: Verify session before sensitive operations\\n❌ **DON\'T**: Pass passwords in command line (visible in history)\\n❌ **DON\'T**: Store tokens in plain text files ### KMS Operations ✅ **DO**: Use context (AAD) for encryption when available\\n✅ **DO**: Rotate KMS keys regularly\\n✅ **DO**: Use hardware-backed keys (WebAuthn, YubiKey) when possible\\n❌ **DON\'T**: Share Age private keys\\n❌ **DON\'T**: Log decrypted data ### Orchestrator ✅ **DO**: Validate workflows in strict mode before production\\n✅ **DO**: Monitor task status regularly\\n✅ **DO**: Use appropriate data directory permissions (700)\\n❌ **DON\'T**: Run orchestrator as root\\n❌ **DON\'T**: Expose data directory over network shares --- ## FAQ **Q: Why use plugins instead of HTTP API?**\\nA: Plugins are 10x faster, have better Nushell integration, and eliminate HTTP overhead. **Q: Can I use plugins without orchestrator running?**\\nA: `auth` and `kms` work independently. `orch` requires access to orchestrator data directory. **Q: How do I update plugins?**\\nA: Rebuild and re-register: `cargo build --release --all && plugin add target/release/nu_plugin_*` **Q: Are plugins cross-platform?**\\nA: Yes, plugins work on macOS, Linux, and Windows (with appropriate keyring services). **Q: Can I use multiple KMS backends simultaneously?**\\nA: Yes, specify `--backend` flag for each operation. **Q: How do I backup MFA enrollment?**\\nA: Save backup codes securely (password manager, encrypted file). QR code can be re-scanned. --- ## Related Documentation - **Security System**: `docs/architecture/ADR-009-security-system-complete.md`\\n- **JWT Auth**: `docs/architecture/JWT_AUTH_IMPLEMENTATION.md`\\n- **Config Encryption**: `docs/user/CONFIG_ENCRYPTION_GUIDE.md`\\n- **RustyVault Integration**: `RUSTYVAULT_INTEGRATION_SUMMARY.md`\\n- **MFA Implementation**: `docs/architecture/MFA_IMPLEMENTATION_SUMMARY.md` --- **Version**: 1.0.0\\n**Last Updated**: 2025-10-09\\n**Maintained By**: Platform Team","breadcrumbs":"NuShell Plugins Guide » Build from Source","id":"1623","title":"Build from Source"},"1624":{"body":"For complete documentation on Nushell plugins including installation, configuration, and advanced usage, see: Complete Guide : Plugin Integration Guide (1500+ lines) Quick Reference : Nushell Plugins Guide","breadcrumbs":"NuShell Plugins System » Nushell Plugins Integration (v1.0.0) - See detailed guide for complete reference","id":"1624","title":"Nushell Plugins Integration (v1.0.0) - See detailed guide for complete reference"},"1625":{"body":"Native Nushell plugins eliminate HTTP overhead and provide direct Rust-to-Nushell integration for critical platform operations.","breadcrumbs":"NuShell Plugins System » Overview","id":"1625","title":"Overview"},"1626":{"body":"Plugin Operation HTTP Latency Plugin Latency Speedup nu_plugin_kms Encrypt (RustyVault) ~50ms ~5ms 10x nu_plugin_kms Decrypt (RustyVault) ~50ms ~5ms 10x nu_plugin_orchestrator Status query ~30ms ~1ms 30x nu_plugin_auth Verify session ~50ms ~10ms 5x","breadcrumbs":"NuShell Plugins System » Performance Improvements","id":"1626","title":"Performance Improvements"},"1627":{"body":"Authentication Plugin (nu_plugin_auth) JWT login/logout with password prompts MFA enrollment (TOTP, WebAuthn) Session management OS-native keyring integration KMS Plugin (nu_plugin_kms) Multiple backend support (RustyVault, Age, Cosmian, AWS KMS, Vault) 10x faster encryption/decryption Context-based encryption (AAD support) Orchestrator Plugin (nu_plugin_orchestrator) Direct file-based operations (no HTTP) 30-50x faster status queries KCL workflow validation","breadcrumbs":"NuShell Plugins System » Three Native Plugins","id":"1627","title":"Three Native Plugins"},"1628":{"body":"# Authentication\\nauth login admin\\nauth verify\\nauth mfa enroll totp # KMS Operations\\nkms encrypt \\"data\\"\\nkms decrypt \\"vault:v1:abc123...\\" # Orchestrator\\norch status\\norch validate workflows/deploy.k\\norch tasks --status running","breadcrumbs":"NuShell Plugins System » Quick Commands","id":"1628","title":"Quick Commands"},"1629":{"body":"cd provisioning/core/plugins/nushell-plugins\\ncargo build --release --all # Register with Nushell\\nplugin add target/release/nu_plugin_auth\\nplugin add target/release/nu_plugin_kms\\nplugin add target/release/nu_plugin_orchestrator","breadcrumbs":"NuShell Plugins System » Installation","id":"1629","title":"Installation"},"163":{"body":"First, run in check mode to see what would happen: # Check mode - no actual changes\\nprovisioning server create --infra my-infra --check # Expected output:\\n# ✓ Validation passed\\n# ⚠ Check mode: No changes will be made\\n# # Would create:\\n# - Server: dev-server-01 (2 cores, 4GB RAM, 50GB disk)","breadcrumbs":"First Deployment » Step 3: Create Server (Check Mode)","id":"163","title":"Step 3: Create Server (Check Mode)"},"1630":{"body":"✅ 10x faster KMS operations (5ms vs 50ms) ✅ 30-50x faster orchestrator queries (1ms vs 30-50ms) ✅ Native Nushell integration with data structures and pipelines ✅ Offline capability (KMS with Age, orchestrator local ops) ✅ OS-native keyring for secure token storage See Plugin Integration Guide for complete information.","breadcrumbs":"NuShell Plugins System » Benefits","id":"1630","title":"Benefits"},"1631":{"body":"","breadcrumbs":"Plugin Usage Guide » Provisioning Plugins Usage Guide","id":"1631","title":"Provisioning Plugins Usage Guide"},"1632":{"body":"Three high-performance Nushell plugins have been integrated into the provisioning system to provide 10-50x performance improvements over HTTP-based operations: nu_plugin_auth - JWT authentication with system keyring integration nu_plugin_kms - Multi-backend KMS encryption nu_plugin_orchestrator - Local orchestrator operations","breadcrumbs":"Plugin Usage Guide » Overview","id":"1632","title":"Overview"},"1633":{"body":"","breadcrumbs":"Plugin Usage Guide » Installation","id":"1633","title":"Installation"},"1634":{"body":"Nushell 0.107.1 or later All plugins are pre-compiled in provisioning/core/plugins/nushell-plugins/","breadcrumbs":"Plugin Usage Guide » Prerequisites","id":"1634","title":"Prerequisites"},"1635":{"body":"Run the installation script in a new Nushell session: nu provisioning/core/plugins/install-and-register.nu This will: Copy plugins to ~/.local/share/nushell/plugins/ Register plugins with Nushell Verify installation","breadcrumbs":"Plugin Usage Guide » Quick Install","id":"1635","title":"Quick Install"},"1636":{"body":"If the script doesn\'t work, run these commands: # Copy plugins\\ncp provisioning/core/plugins/nushell-plugins/nu_plugin_auth/target/release/nu_plugin_auth ~/.local/share/nushell/plugins/\\ncp provisioning/core/plugins/nushell-plugins/nu_plugin_kms/target/release/nu_plugin_kms ~/.local/share/nushell/plugins/\\ncp provisioning/core/plugins/nushell-plugins/nu_plugin_orchestrator/target/release/nu_plugin_orchestrator ~/.local/share/nushell/plugins/ chmod +x ~/.local/share/nushell/plugins/nu_plugin_* # Register with Nushell (run in a fresh session)\\nplugin add ~/.local/share/nushell/plugins/nu_plugin_auth\\nplugin add ~/.local/share/nushell/plugins/nu_plugin_kms\\nplugin add ~/.local/share/nushell/plugins/nu_plugin_orchestrator","breadcrumbs":"Plugin Usage Guide » Manual Installation","id":"1636","title":"Manual Installation"},"1637":{"body":"","breadcrumbs":"Plugin Usage Guide » Usage","id":"1637","title":"Usage"},"1638":{"body":"10x faster than HTTP fallback Login provisioning auth login  [password] # Examples\\nprovisioning auth login admin\\nprovisioning auth login admin mypassword\\nprovisioning auth login --url http://localhost:8081 admin Verify Token provisioning auth verify [--local] # Examples\\nprovisioning auth verify\\nprovisioning auth verify --local Logout provisioning auth logout # Example\\nprovisioning auth logout List Sessions provisioning auth sessions [--active] # Examples\\nprovisioning auth sessions\\nprovisioning auth sessions --active","breadcrumbs":"Plugin Usage Guide » Authentication Plugin","id":"1638","title":"Authentication Plugin"},"1639":{"body":"10x faster than HTTP fallback Supports multiple backends: RustyVault, Age, AWS KMS, HashiCorp Vault, Cosmian Encrypt Data provisioning kms encrypt  [--backend ] [--key ] # Examples\\nprovisioning kms encrypt \\"secret-data\\"\\nprovisioning kms encrypt \\"secret\\" --backend age\\nprovisioning kms encrypt \\"secret\\" --backend rustyvault --key my-key Decrypt Data provisioning kms decrypt  [--backend ] [--key ] # Examples\\nprovisioning kms decrypt $encrypted_data\\nprovisioning kms decrypt $encrypted --backend age KMS Status provisioning kms status # Output shows current backend and availability List Backends provisioning kms list-backends # Shows all available KMS backends","breadcrumbs":"Plugin Usage Guide » KMS Plugin","id":"1639","title":"KMS Plugin"},"164":{"body":"If check mode looks good, create the server: # Create server\\nprovisioning server create --infra my-infra # Expected output:\\n# ✓ Creating server: dev-server-01\\n# ✓ Server created successfully\\n# ✓ IP Address: 192.168.1.100\\n# ✓ SSH access: ssh user@192.168.1.100","breadcrumbs":"First Deployment » Step 4: Create Server (Real)","id":"164","title":"Step 4: Create Server (Real)"},"1640":{"body":"30x faster than HTTP fallback Local file-based orchestration without network overhead. Check Status provisioning orch status [--data-dir ] # Examples\\nprovisioning orch status\\nprovisioning orch status --data-dir /custom/data List Tasks provisioning orch tasks [--status ] [--limit ] [--data-dir ] # Examples\\nprovisioning orch tasks\\nprovisioning orch tasks --status pending\\nprovisioning orch tasks --status running --limit 10 Validate Workflow provisioning orch validate  [--strict] # Examples\\nprovisioning orch validate workflows/deployment.k\\nprovisioning orch validate workflows/deployment.k --strict Submit Workflow provisioning orch submit  [--priority <0-100>] [--check] # Examples\\nprovisioning orch submit workflows/deployment.k\\nprovisioning orch submit workflows/critical.k --priority 90\\nprovisioning orch submit workflows/test.k --check Monitor Task provisioning orch monitor  [--once] [--interval ] [--timeout ] # Examples\\nprovisioning orch monitor task-123\\nprovisioning orch monitor task-123 --once\\nprovisioning orch monitor task-456 --interval 5000 --timeout 600","breadcrumbs":"Plugin Usage Guide » Orchestrator Plugin","id":"1640","title":"Orchestrator Plugin"},"1641":{"body":"Check which plugins are installed: provisioning plugin status # Output:\\n# Provisioning Plugins Status\\n# ============================\\n# [OK] nu_plugin_auth - JWT authentication with keyring\\n# [OK] nu_plugin_kms - Multi-backend encryption\\n# [OK] nu_plugin_orchestrator - Local orchestrator (30x faster)\\n#\\n# All plugins loaded - using native high-performance mode","breadcrumbs":"Plugin Usage Guide » Plugin Status","id":"1641","title":"Plugin Status"},"1642":{"body":"provisioning plugin test # Runs quick tests on all installed plugins\\n# Output shows which plugins are responding","breadcrumbs":"Plugin Usage Guide » Testing Plugins","id":"1642","title":"Testing Plugins"},"1643":{"body":"provisioning plugin list # Shows all provisioning plugins registered with Nushell","breadcrumbs":"Plugin Usage Guide » List Registered Plugins","id":"1643","title":"List Registered Plugins"},"1644":{"body":"Operation With Plugin HTTP Fallback Speedup Auth verify ~10ms ~50ms 5x Auth login ~15ms ~100ms 7x KMS encrypt ~5-8ms ~50ms 10x KMS decrypt ~5-8ms ~50ms 10x Orch status ~1-5ms ~30ms 30x Orch tasks list ~2-10ms ~50ms 25x","breadcrumbs":"Plugin Usage Guide » Performance Comparison","id":"1644","title":"Performance Comparison"},"1645":{"body":"If plugins are not installed or fail to load, all commands automatically fall back to HTTP-based operations: # With plugins installed (fast)\\n$ provisioning auth verify\\nToken is valid # Without plugins (slower, but functional)\\n$ provisioning auth verify\\n[HTTP fallback mode]\\nToken is valid (slower) This ensures the system remains functional even if plugins aren\'t available.","breadcrumbs":"Plugin Usage Guide » Graceful Fallback","id":"1645","title":"Graceful Fallback"},"1646":{"body":"","breadcrumbs":"Plugin Usage Guide » Troubleshooting","id":"1646","title":"Troubleshooting"},"1647":{"body":"Make sure you: Have a fresh Nushell session Ran plugin add for all three plugins The plugin files are executable: chmod +x ~/.local/share/nushell/plugins/nu_plugin_*","breadcrumbs":"Plugin Usage Guide » Plugins not found after installation","id":"1647","title":"Plugins not found after installation"},"1648":{"body":"If you see \\"command not found\\" when running provisioning auth login, the auth plugin is not loaded. Run: plugin list | grep nu_plugin If you don\'t see the plugins, register them: plugin add ~/.local/share/nushell/plugins/nu_plugin_auth\\nplugin add ~/.local/share/nushell/plugins/nu_plugin_kms\\nplugin add ~/.local/share/nushell/plugins/nu_plugin_orchestrator","breadcrumbs":"Plugin Usage Guide » \\"Command not found\\" errors","id":"1648","title":"\\"Command not found\\" errors"},"1649":{"body":"Check the plugin logs: provisioning plugin test If a plugin fails, the system will automatically fall back to HTTP mode.","breadcrumbs":"Plugin Usage Guide » Plugins crash or are unresponsive","id":"1649","title":"Plugins crash or are unresponsive"},"165":{"body":"Check server status: # List all servers\\nprovisioning server list # Get detailed server info\\nprovisioning server info dev-server-01 # SSH to server (optional)\\nprovisioning server ssh dev-server-01","breadcrumbs":"First Deployment » Step 5: Verify Server","id":"165","title":"Step 5: Verify Server"},"1650":{"body":"All plugin commands are integrated into the main provisioning CLI: # Shortcuts available\\nprovisioning auth login admin # Full command\\nprovisioning login admin # Alias provisioning kms encrypt secret # Full command\\nprovisioning encrypt secret # Alias provisioning orch status # Full command\\nprovisioning orch-status # Alias","breadcrumbs":"Plugin Usage Guide » Integration with Provisioning CLI","id":"1650","title":"Integration with Provisioning CLI"},"1651":{"body":"","breadcrumbs":"Plugin Usage Guide » Advanced Configuration","id":"1651","title":"Advanced Configuration"},"1652":{"body":"For orchestrator operations, specify custom data directory: provisioning orch status --data-dir /custom/orchestrator/data\\nprovisioning orch tasks --data-dir /custom/orchestrator/data","breadcrumbs":"Plugin Usage Guide » Custom Data Directory","id":"1652","title":"Custom Data Directory"},"1653":{"body":"For auth operations with custom endpoint: provisioning auth login admin --url http://custom-auth-server:8081\\nprovisioning auth verify --url http://custom-auth-server:8081","breadcrumbs":"Plugin Usage Guide » Custom Auth URL","id":"1653","title":"Custom Auth URL"},"1654":{"body":"Specify which KMS backend to use: # Use Age encryption\\nprovisioning kms encrypt \\"data\\" --backend age # Use RustyVault\\nprovisioning kms encrypt \\"data\\" --backend rustyvault # Use AWS KMS\\nprovisioning kms encrypt \\"data\\" --backend aws # Decrypt with same backend\\nprovisioning kms decrypt $encrypted --backend age","breadcrumbs":"Plugin Usage Guide » KMS Backend Selection","id":"1654","title":"KMS Backend Selection"},"1655":{"body":"If you need to rebuild plugins: cd provisioning/core/plugins/nushell-plugins # Build auth plugin\\ncd nu_plugin_auth && cargo build --release && cd .. # Build KMS plugin\\ncd nu_plugin_kms && cargo build --release && cd .. # Build orchestrator plugin\\ncd nu_plugin_orchestrator && cargo build --release && cd .. # Run install script\\ncd ../..\\nnu install-and-register.nu","breadcrumbs":"Plugin Usage Guide » Building Plugins from Source","id":"1655","title":"Building Plugins from Source"},"1656":{"body":"The plugins follow Nushell\'s plugin protocol: Plugin Binary : Compiled Rust binary in target/release/ Registration : Via plugin add command IPC : Communication via Nushell\'s JSON protocol Fallback : HTTP API fallback if plugins unavailable","breadcrumbs":"Plugin Usage Guide » Architecture","id":"1656","title":"Architecture"},"1657":{"body":"Auth tokens are stored in system keyring (Keychain/Credential Manager/Secret Service) KMS keys are protected by the selected backend\'s security Orchestrator operations are local file-based (no network exposure) All operations are logged in provisioning audit logs","breadcrumbs":"Plugin Usage Guide » Security Notes","id":"1657","title":"Security Notes"},"1658":{"body":"For issues or questions: Check plugin status: provisioning plugin test Review logs: provisioning logs or /var/log/provisioning/ Test HTTP fallback by temporarily unregistering plugins Contact the provisioning team with plugin test output","breadcrumbs":"Plugin Usage Guide » Support","id":"1658","title":"Support"},"1659":{"body":"Status : Production Ready Date : 2025-11-19 Version : 1.0.0","breadcrumbs":"Secrets Management Guide » Secrets Management System - Configuration Guide","id":"1659","title":"Secrets Management System - Configuration Guide"},"166":{"body":"Install a task service on the server: # Check mode first\\nprovisioning taskserv create kubernetes --infra my-infra --check # Expected output:\\n# ✓ Validation passed\\n# ⚠ Check mode: No changes will be made\\n#\\n# Would install:\\n# - Kubernetes v1.28.0\\n# - Required dependencies: containerd, etcd\\n# - On servers: dev-server-01","breadcrumbs":"First Deployment » Step 6: Install Kubernetes (Check Mode)","id":"166","title":"Step 6: Install Kubernetes (Check Mode)"},"1660":{"body":"The provisioning system supports secure SSH key retrieval from multiple secret sources, eliminating hardcoded filesystem dependencies and enabling enterprise-grade security. SSH keys are retrieved from configured secret sources (SOPS, KMS, RustyVault) with automatic fallback to local-dev mode for development environments.","breadcrumbs":"Secrets Management Guide » Overview","id":"1660","title":"Overview"},"1661":{"body":"","breadcrumbs":"Secrets Management Guide » Secret Sources","id":"1661","title":"Secret Sources"},"1662":{"body":"Age-based encrypted secrets file with YAML structure. Pros : ✅ Age encryption (modern, performant) ✅ Easy to version in Git (encrypted) ✅ No external services required ✅ Simple YAML structure Cons : ❌ Requires Age key management ❌ No key rotation automation Environment Variables : PROVISIONING_SECRET_SOURCE=sops\\nPROVISIONING_SOPS_ENABLED=true\\nPROVISIONING_SOPS_SECRETS_FILE=/path/to/secrets.enc.yaml\\nPROVISIONING_SOPS_AGE_KEY_FILE=$HOME/.age/provisioning\\n```plaintext **Secrets File Structure** (provisioning/secrets.enc.yaml): ```yaml\\n# Encrypted with sops\\nssh: web-01: ubuntu: /path/to/id_rsa root: /path/to/root_id_rsa db-01: postgres: /path/to/postgres_id_rsa\\n```plaintext **Setup Instructions**: ```bash\\n# 1. Install sops and age\\nbrew install sops age # 2. Generate Age key (store securely!)\\nage-keygen -o $HOME/.age/provisioning # 3. Create encrypted secrets file\\ncat > secrets.yaml << \'EOF\'\\nssh: web-01: ubuntu: ~/.ssh/provisioning_web01 db-01: postgres: ~/.ssh/provisioning_db01\\nEOF # 4. Encrypt with sops\\nsops -e -i secrets.yaml # 5. Rename to enc version\\nmv secrets.yaml provisioning/secrets.enc.yaml # 6. Configure environment\\nexport PROVISIONING_SECRET_SOURCE=sops\\nexport PROVISIONING_SOPS_SECRETS_FILE=$(pwd)/provisioning/secrets.enc.yaml\\nexport PROVISIONING_SOPS_AGE_KEY_FILE=$HOME/.age/provisioning\\n```plaintext ### 2. KMS (Key Management Service) AWS KMS or compatible key management service. **Pros**: - ✅ Cloud-native security\\n- ✅ Automatic key rotation\\n- ✅ Audit logging built-in\\n- ✅ High availability **Cons**: - ❌ Requires AWS account/credentials\\n- ❌ API calls add latency (~50ms)\\n- ❌ Cost per API call **Environment Variables**: ```bash\\nPROVISIONING_SECRET_SOURCE=kms\\nPROVISIONING_KMS_ENABLED=true\\nPROVISIONING_KMS_REGION=us-east-1\\n```plaintext **Secret Storage Pattern**: ```plaintext\\nprovisioning/ssh-keys/{hostname}/{username}\\n```plaintext **Setup Instructions**: ```bash\\n# 1. Create KMS key (one-time)\\naws kms create-key \\\\ --description \\"Provisioning SSH Keys\\" \\\\ --region us-east-1 # 2. Store SSH keys in Secrets Manager\\naws secretsmanager create-secret \\\\ --name provisioning/ssh-keys/web-01/ubuntu \\\\ --secret-string \\"$(cat ~/.ssh/provisioning_web01)\\" \\\\ --region us-east-1 # 3. Configure environment\\nexport PROVISIONING_SECRET_SOURCE=kms\\nexport PROVISIONING_KMS_REGION=us-east-1 # 4. Ensure AWS credentials available\\nexport AWS_PROFILE=provisioning\\n# or\\nexport AWS_ACCESS_KEY_ID=...\\nexport AWS_SECRET_ACCESS_KEY=...\\n```plaintext ### 3. RustyVault (Hashicorp Vault-Compatible) Self-hosted or managed Vault instance for secrets. **Pros**: - ✅ Self-hosted option\\n- ✅ Fine-grained access control\\n- ✅ Multiple authentication methods\\n- ✅ Easy key rotation **Cons**: - ❌ Requires Vault instance\\n- ❌ More operational overhead\\n- ❌ Network latency **Environment Variables**: ```bash\\nPROVISIONING_SECRET_SOURCE=vault\\nPROVISIONING_VAULT_ENABLED=true\\nPROVISIONING_VAULT_ADDRESS=http://localhost:8200\\nPROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ...\\n```plaintext **Secret Storage Pattern**: ```plaintext\\nGET /v1/secret/ssh-keys/{hostname}/{username}\\n# Returns: {\\"key_content\\": \\"-----BEGIN OPENSSH PRIVATE KEY-----...\\"}\\n```plaintext **Setup Instructions**: ```bash\\n# 1. Start Vault (if not already running)\\ndocker run -p 8200:8200 \\\\ -e VAULT_DEV_ROOT_TOKEN_ID=provisioning \\\\ vault server -dev # 2. Create KV v2 mount (if not exists)\\nvault secrets enable -version=2 -path=secret kv # 3. Store SSH key\\nvault kv put secret/ssh-keys/web-01/ubuntu \\\\ key_content=@~/.ssh/provisioning_web01 # 4. Configure environment\\nexport PROVISIONING_SECRET_SOURCE=vault\\nexport PROVISIONING_VAULT_ADDRESS=http://localhost:8200\\nexport PROVISIONING_VAULT_TOKEN=provisioning # 5. Create AppRole for production\\nvault auth enable approle\\nvault write auth/approle/role/provisioning \\\\ token_ttl=1h \\\\ token_max_ttl=4h\\nvault read auth/approle/role/provisioning/role-id\\nvault write -f auth/approle/role/provisioning/secret-id\\n```plaintext ### 4. Local-Dev (Fallback) Local filesystem SSH keys (development only). **Pros**: - ✅ No setup required\\n- ✅ Fast (local filesystem)\\n- ✅ Works offline **Cons**: - ❌ NOT for production\\n- ❌ Hardcoded filesystem dependency\\n- ❌ No key rotation **Environment Variables**: ```bash\\nPROVISIONING_ENVIRONMENT=local-dev\\n```plaintext **Behavior**: Standard paths checked (in order): 1. `$HOME/.ssh/id_rsa`\\n2. `$HOME/.ssh/id_ed25519`\\n3. `$HOME/.ssh/provisioning`\\n4. `$HOME/.ssh/provisioning_rsa` ## Auto-Detection Logic When `PROVISIONING_SECRET_SOURCE` is not explicitly set, the system auto-detects in this order: ```plaintext\\n1. PROVISIONING_SOPS_ENABLED=true or PROVISIONING_SOPS_SECRETS_FILE set? → Use SOPS\\n2. PROVISIONING_KMS_ENABLED=true or PROVISIONING_KMS_REGION set? → Use KMS\\n3. PROVISIONING_VAULT_ENABLED=true or both VAULT_ADDRESS and VAULT_TOKEN set? → Use Vault\\n4. Otherwise → Use local-dev (with warnings in production environments)\\n```plaintext ## Configuration Matrix | Secret Source | Env Variables | Enabled in |\\n|---|---|---|\\n| **SOPS** | `PROVISIONING_SOPS_*` | Development, Staging, Production |\\n| **KMS** | `PROVISIONING_KMS_*` | Staging, Production (with AWS) |\\n| **Vault** | `PROVISIONING_VAULT_*` | Development, Staging, Production |\\n| **Local-dev** | `PROVISIONING_ENVIRONMENT=local-dev` | Development only | ## Production Recommended Setup ### Minimal Setup (Single Source) ```bash\\n# Using Vault (recommended for self-hosted)\\nexport PROVISIONING_SECRET_SOURCE=vault\\nexport PROVISIONING_VAULT_ADDRESS=https://vault.example.com:8200\\nexport PROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ...\\nexport PROVISIONING_ENVIRONMENT=production\\n```plaintext ### Enhanced Setup (Fallback Chain) ```bash\\n# Primary: Vault\\nexport PROVISIONING_VAULT_ADDRESS=https://vault.primary.com:8200\\nexport PROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ... # Fallback: SOPS\\nexport PROVISIONING_SOPS_SECRETS_FILE=/etc/provisioning/secrets.enc.yaml\\nexport PROVISIONING_SOPS_AGE_KEY_FILE=/etc/provisioning/.age/key # Environment\\nexport PROVISIONING_ENVIRONMENT=production\\nexport PROVISIONING_SECRET_SOURCE=vault # Explicit: use Vault first\\n```plaintext ### High-Availability Setup ```bash\\n# Use KMS (managed service)\\nexport PROVISIONING_SECRET_SOURCE=kms\\nexport PROVISIONING_KMS_REGION=us-east-1\\nexport AWS_PROFILE=provisioning-admin # Or use Vault with HA\\nexport PROVISIONING_VAULT_ADDRESS=https://vault-ha.example.com:8200\\nexport PROVISIONING_VAULT_NAMESPACE=provisioning\\nexport PROVISIONING_ENVIRONMENT=production\\n```plaintext ## Validation & Testing ### Check Configuration ```bash\\n# Nushell\\nprovisioning secrets status # Show secret source and configuration\\nprovisioning secrets validate # Detailed diagnostics\\nprovisioning secrets diagnose\\n```plaintext ### Test SSH Key Retrieval ```bash\\n# Test specific host/user\\nprovisioning secrets get-key web-01 ubuntu # Test all configured hosts\\nprovisioning secrets validate-all # Dry-run SSH with retrieved key\\nprovisioning ssh --test-key web-01 ubuntu\\n```plaintext ## Migration Path ### From Local-Dev to SOPS ```bash\\n# 1. Create SOPS secrets file with existing keys\\ncat > secrets.yaml << \'EOF\'\\nssh: web-01: ubuntu: ~/.ssh/provisioning_web01 db-01: postgres: ~/.ssh/provisioning_db01\\nEOF # 2. Encrypt with Age\\nsops -e -i secrets.yaml # 3. Move to repo\\nmv secrets.yaml provisioning/secrets.enc.yaml # 4. Update environment\\nexport PROVISIONING_SECRET_SOURCE=sops\\nexport PROVISIONING_SOPS_SECRETS_FILE=$(pwd)/provisioning/secrets.enc.yaml\\nexport PROVISIONING_SOPS_AGE_KEY_FILE=$HOME/.age/provisioning\\n```plaintext ### From SOPS to Vault ```bash\\n# 1. Decrypt SOPS file\\nsops -d provisioning/secrets.enc.yaml > /tmp/secrets.yaml # 2. Import to Vault\\nvault kv put secret/ssh-keys/web-01/ubuntu key_content=@~/.ssh/provisioning_web01 # 3. Update environment\\nexport PROVISIONING_SECRET_SOURCE=vault\\nexport PROVISIONING_VAULT_ADDRESS=http://vault.example.com:8200\\nexport PROVISIONING_VAULT_TOKEN=hvs.CAESIAoICQ... # 4. Validate retrieval works\\nprovisioning secrets validate-all\\n```plaintext ## Security Best Practices ### 1. Never Commit Secrets ```bash\\n# Add to .gitignore\\necho \\"provisioning/secrets.enc.yaml\\" >> .gitignore\\necho \\".age/provisioning\\" >> .gitignore\\necho \\".vault-token\\" >> .gitignore\\n```plaintext ### 2. Rotate Keys Regularly ```bash\\n# SOPS: Rotate Age key\\nage-keygen -o ~/.age/provisioning.new\\n# Update all secrets with new key # KMS: Enable automatic rotation\\naws kms enable-key-rotation --key-id alias/provisioning # Vault: Set TTL on secrets\\nvault write -f secret/metadata/ssh-keys/web-01/ubuntu \\\\ delete_version_after=2160h # 90 days\\n```plaintext ### 3. Restrict Access ```bash\\n# SOPS: Protect Age key\\nchmod 600 ~/.age/provisioning # KMS: Restrict IAM permissions\\naws iam put-user-policy --user-name provisioning \\\\ --policy-name ProvisioningSecretsAccess \\\\ --policy-document file://kms-policy.json # Vault: Use AppRole for applications\\nvault write auth/approle/role/provisioning \\\\ token_ttl=1h \\\\ secret_id_ttl=30m\\n```plaintext ### 4. Audit Logging ```bash\\n# KMS: Enable CloudTrail\\naws cloudtrail put-event-selectors \\\\ --trail-name provisioning-trail \\\\ --event-selectors ReadWriteType=All # Vault: Check audit logs\\nvault audit list # SOPS: Version control (encrypted)\\ngit log -p provisioning/secrets.enc.yaml\\n```plaintext ## Troubleshooting ### SOPS Issues ```bash\\n# Test Age decryption\\nsops -d provisioning/secrets.enc.yaml # Verify Age key\\nage-keygen -l ~/.age/provisioning # Regenerate if needed\\nrm ~/.age/provisioning\\nage-keygen -o ~/.age/provisioning\\n```plaintext ### KMS Issues ```bash\\n# Test AWS credentials\\naws sts get-caller-identity # Check KMS key permissions\\naws kms describe-key --key-id alias/provisioning # List secrets\\naws secretsmanager list-secrets --filters Name=name,Values=provisioning\\n```plaintext ### Vault Issues ```bash\\n# Check Vault status\\nvault status # Test authentication\\nvault token lookup # List secrets\\nvault kv list secret/ssh-keys/ # Check audit logs\\nvault audit list\\nvault read sys/audit\\n```plaintext ## FAQ **Q: Can I use multiple secret sources simultaneously?**\\nA: Yes, configure multiple sources and set `PROVISIONING_SECRET_SOURCE` to specify primary. If primary fails, manual fallback to secondary is supported. **Q: What happens if secret retrieval fails?**\\nA: System logs the error and fails fast. No automatic fallback to local filesystem (for security). **Q: Can I cache SSH keys?**\\nA: Currently not, keys are retrieved fresh for each operation. Use local caching at OS level (ssh-agent) if needed. **Q: How do I rotate keys?**\\nA: Update the secret in your configured source (SOPS/KMS/Vault) and retrieve fresh on next operation. **Q: Is local-dev mode secure?**\\nA: No - it\'s development only. Production requires SOPS/KMS/Vault. ## Architecture ```plaintext\\nSSH Operation ↓\\nSecretsManager (Nushell/Rust) ↓\\n[Detect Source] ↓\\n┌─────────────────────────────────────┐\\n│ SOPS KMS Vault LocalDev\\n│ (Encrypted (AWS KMS (Self- (Filesystem\\n│ Secrets) Service) Hosted) Dev Only)\\n│\\n└─────────────────────────────────────┘ ↓\\nReturn SSH Key Path/Content ↓\\nSSH Operation Completes\\n```plaintext ## Integration with SSH Utilities SSH operations automatically use secrets manager: ```nushell\\n# Automatic secret retrieval\\nssh-cmd-smart $settings $server false \\"command\\" $ip\\n# Internally:\\n# 1. Determine secret source\\n# 2. Retrieve SSH key for server.installer_user@ip\\n# 3. Execute SSH with retrieved key\\n# 4. Cleanup sensitive data # Batch operations also integrate\\nssh-batch-execute $servers $settings \\"command\\"\\n# Per-host: Retrieves key → executes → cleans up\\n```plaintext --- **For Support**: See `docs/user/TROUBLESHOOTING_GUIDE.md`\\n**For Integration**: See `provisioning/core/nulib/lib_provisioning/platform/secrets.nu`","breadcrumbs":"Secrets Management Guide » 1. SOPS (Secrets Operations)","id":"1662","title":"1. SOPS (Secrets Operations)"},"1663":{"body":"","breadcrumbs":"Auth Quick Reference » Auth Quick Reference","id":"1663","title":"Auth Quick Reference"},"1664":{"body":"","breadcrumbs":"Config Encryption Quick Reference » Config Encryption Quick Reference","id":"1664","title":"Config Encryption Quick Reference"},"1665":{"body":"A unified Key Management Service for the Provisioning platform with support for multiple backends. Source : provisioning/platform/kms-service/","breadcrumbs":"KMS Service » KMS Service - Key Management Service","id":"1665","title":"KMS Service - Key Management Service"},"1666":{"body":"Age : Fast, offline encryption (development) RustyVault : Self-hosted Vault-compatible API Cosmian KMS : Enterprise-grade with confidential computing AWS KMS : Cloud-native key management HashiCorp Vault : Enterprise secrets management","breadcrumbs":"KMS Service » Supported Backends","id":"1666","title":"Supported Backends"},"1667":{"body":"┌─────────────────────────────────────────────────────────┐\\n│ KMS Service │\\n├─────────────────────────────────────────────────────────┤\\n│ REST API (Axum) │\\n│ ├─ /api/v1/kms/encrypt POST │\\n│ ├─ /api/v1/kms/decrypt POST │\\n│ ├─ /api/v1/kms/generate-key POST │\\n│ ├─ /api/v1/kms/status GET │\\n│ └─ /api/v1/kms/health GET │\\n├─────────────────────────────────────────────────────────┤\\n│ Unified KMS Service Interface │\\n├─────────────────────────────────────────────────────────┤\\n│ Backend Implementations │\\n│ ├─ Age Client (local files) │\\n│ ├─ RustyVault Client (self-hosted) │\\n│ └─ Cosmian KMS Client (enterprise) │\\n└─────────────────────────────────────────────────────────┘\\n```plaintext ## Quick Start ### Development Setup (Age) ```bash\\n# 1. Generate Age keys\\nmkdir -p ~/.config/provisioning/age\\nage-keygen -o ~/.config/provisioning/age/private_key.txt\\nage-keygen -y ~/.config/provisioning/age/private_key.txt > ~/.config/provisioning/age/public_key.txt # 2. Set environment\\nexport PROVISIONING_ENV=dev # 3. Start KMS service\\ncd provisioning/platform/kms-service\\ncargo run --bin kms-service\\n```plaintext ### Production Setup (Cosmian) ```bash\\n# Set environment variables\\nexport PROVISIONING_ENV=prod\\nexport COSMIAN_KMS_URL=https://your-kms.example.com\\nexport COSMIAN_API_KEY=your-api-key-here # Start KMS service\\ncargo run --bin kms-service\\n```plaintext ## REST API Examples ### Encrypt Data ```bash\\ncurl -X POST http://localhost:8082/api/v1/kms/encrypt \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{ \\"plaintext\\": \\"SGVsbG8sIFdvcmxkIQ==\\", \\"context\\": \\"env=prod,service=api\\" }\'\\n```plaintext ### Decrypt Data ```bash\\ncurl -X POST http://localhost:8082/api/v1/kms/decrypt \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{ \\"ciphertext\\": \\"...\\", \\"context\\": \\"env=prod,service=api\\" }\'\\n```plaintext ## Nushell CLI Integration ```bash\\n# Encrypt data\\n\\"secret-data\\" | kms encrypt\\n\\"api-key\\" | kms encrypt --context \\"env=prod,service=api\\" # Decrypt data\\n$ciphertext | kms decrypt # Generate data key (Cosmian only)\\nkms generate-key # Check service status\\nkms status\\nkms health # Encrypt/decrypt files\\nkms encrypt-file config.yaml\\nkms decrypt-file config.yaml.enc\\n```plaintext ## Backend Comparison | Feature | Age | RustyVault | Cosmian KMS | AWS KMS | Vault |\\n|---------|-----|------------|-------------|---------|-------|\\n| **Setup** | Simple | Self-hosted | Server setup | AWS account | Enterprise |\\n| **Speed** | Very fast | Fast | Fast | Fast | Fast |\\n| **Network** | No | Yes | Yes | Yes | Yes |\\n| **Key Rotation** | Manual | Automatic | Automatic | Automatic | Automatic |\\n| **Data Keys** | No | Yes | Yes | Yes | Yes |\\n| **Audit Logging** | No | Yes | Full | Full | Full |\\n| **Confidential** | No | No | Yes (SGX/SEV) | No | No |\\n| **License** | MIT | Apache 2.0 | Proprietary | Proprietary | BSL/Enterprise |\\n| **Cost** | Free | Free | Paid | Paid | Paid |\\n| **Use Case** | Dev/Test | Self-hosted | Privacy | AWS Cloud | Enterprise | ## Integration Points 1. **Config Encryption** (SOPS Integration)\\n2. **Dynamic Secrets** (Provider API Keys)\\n3. **SSH Key Management**\\n4. **Orchestrator** (Workflow Data)\\n5. **Control Center** (Audit Logs) ## Deployment ### Docker ```dockerfile\\nFROM rust:1.70 as builder\\nWORKDIR /app\\nCOPY . .\\nRUN cargo build --release FROM debian:bookworm-slim\\nRUN apt-get update && \\\\ apt-get install -y ca-certificates && \\\\ rm -rf /var/lib/apt/lists/*\\nCOPY --from=builder /app/target/release/kms-service /usr/local/bin/\\nENTRYPOINT [\\"kms-service\\"]\\n```plaintext ### Kubernetes ```yaml\\napiVersion: apps/v1\\nkind: Deployment\\nmetadata: name: kms-service\\nspec: replicas: 2 template: spec: containers: - name: kms-service image: provisioning/kms-service:latest env: - name: PROVISIONING_ENV value: \\"prod\\" - name: COSMIAN_KMS_URL value: \\"https://kms.example.com\\" ports: - containerPort: 8082\\n```plaintext ## Security Best Practices 1. **Development**: Use Age for dev/test only, never for production secrets\\n2. **Production**: Always use Cosmian KMS with TLS verification enabled\\n3. **API Keys**: Never hardcode, use environment variables\\n4. **Key Rotation**: Enable automatic rotation (90 days recommended)\\n5. **Context Encryption**: Always use encryption context (AAD)\\n6. **Network Access**: Restrict KMS service access with firewall rules\\n7. **Monitoring**: Enable health checks and monitor operation metrics ## Related Documentation - **User Guide**: [KMS Guide](../user/RUSTYVAULT_KMS_GUIDE.md)\\n- **Migration**: [KMS Simplification](../migration/KMS_SIMPLIFICATION.md)","breadcrumbs":"KMS Service » Architecture","id":"1667","title":"Architecture"},"1668":{"body":"Complete guide to using Gitea integration for workspace management, extension distribution, and collaboration. Version: 1.0.0 Last Updated: 2025-10-06","breadcrumbs":"Gitea Integration Guide » Gitea Integration Guide","id":"1668","title":"Gitea Integration Guide"},"1669":{"body":"Overview Setup Workspace Git Integration Workspace Locking Extension Publishing Service Management API Reference Troubleshooting","breadcrumbs":"Gitea Integration Guide » Table of Contents","id":"1669","title":"Table of Contents"},"167":{"body":"Proceed with installation: # Install Kubernetes\\nprovisioning taskserv create kubernetes --infra my-infra --wait # This will:\\n# 1. Check dependencies\\n# 2. Install containerd\\n# 3. Install etcd\\n# 4. Install Kubernetes\\n# 5. Configure and start services # Monitor progress\\nprovisioning workflow monitor ","breadcrumbs":"First Deployment » Step 7: Install Kubernetes (Real)","id":"167","title":"Step 7: Install Kubernetes (Real)"},"1670":{"body":"The Gitea integration provides: Workspace Git Integration : Version control for workspaces Distributed Locking : Prevent concurrent workspace modifications Extension Distribution : Publish and download extensions via releases Collaboration : Share workspaces and extensions across teams Service Management : Deploy and manage local Gitea instance","breadcrumbs":"Gitea Integration Guide » Overview","id":"1670","title":"Overview"},"1671":{"body":"┌─────────────────────────────────────────────────────────┐\\n│ Provisioning System │\\n├─────────────────────────────────────────────────────────┤\\n│ │\\n│ ┌────────────┐ ┌──────────────┐ ┌─────────────────┐ │\\n│ │ Workspace │ │ Extension │ │ Locking │ │\\n│ │ Git │ │ Publishing │ │ (Issues) │ │\\n│ └─────┬──────┘ └──────┬───────┘ └────────┬────────┘ │\\n│ │ │ │ │\\n│ └────────────────┼─────────────────────┘ │\\n│ │ │\\n│ ┌──────▼──────┐ │\\n│ │ Gitea API │ │\\n│ │ Client │ │\\n│ └──────┬──────┘ │\\n│ │ │\\n└─────────────────────────┼────────────────────────────────┘ │ ┌───────▼────────┐ │ Gitea Service │ │ (Local/Remote)│ └────────────────┘\\n```plaintext --- ## Setup ### Prerequisites - **Nushell 0.107.1+**\\n- **Git** installed and configured\\n- **Docker** (for local Gitea deployment) or access to remote Gitea instance\\n- **SOPS** (for encrypted token storage) ### Configuration #### 1. Add Gitea Configuration to KCL Edit your `provisioning/kcl/modes.k` or workspace config: ```kcl\\nimport provisioning.gitea as gitea # Local Docker deployment\\n_gitea_config = gitea.GiteaConfig { mode = \\"local\\" local = gitea.LocalGitea { enabled = True deployment = \\"docker\\" port = 3000 auto_start = True docker = gitea.DockerGitea { image = \\"gitea/gitea:1.21\\" container_name = \\"provisioning-gitea\\" } } auth = gitea.GiteaAuth { token_path = \\"~/.provisioning/secrets/gitea-token.enc\\" username = \\"provisioning\\" }\\n} # Or remote Gitea instance\\n_gitea_remote = gitea.GiteaConfig { mode = \\"remote\\" remote = gitea.RemoteGitea { enabled = True url = \\"https://gitea.example.com\\" api_url = \\"https://gitea.example.com/api/v1\\" } auth = gitea.GiteaAuth { token_path = \\"~/.provisioning/secrets/gitea-token.enc\\" username = \\"myuser\\" }\\n}\\n```plaintext #### 2. Create Gitea Access Token For local Gitea: 1. Start Gitea: `provisioning gitea start`\\n2. Open \\n3. Register admin account\\n4. Go to Settings → Applications → Generate New Token\\n5. Save token to encrypted file: ```bash\\n# Create encrypted token file\\necho \\"your-gitea-token\\" | sops --encrypt /dev/stdin > ~/.provisioning/secrets/gitea-token.enc\\n```plaintext For remote Gitea: 1. Login to your Gitea instance\\n2. Generate personal access token\\n3. Save encrypted as above #### 3. Verify Setup ```bash\\n# Check Gitea status\\nprovisioning gitea status # Validate token\\nprovisioning gitea auth validate # Show current user\\nprovisioning gitea user\\n```plaintext --- ## Workspace Git Integration ### Initialize Workspace with Git When creating a new workspace, enable git integration: ```bash\\n# Initialize new workspace with Gitea\\nprovisioning workspace init my-workspace --git --remote gitea # Or initialize existing workspace\\ncd workspace_my-workspace\\nprovisioning gitea workspace init . my-workspace --remote gitea\\n```plaintext This will: 1. Initialize git repository in workspace\\n2. Create repository on Gitea (`workspaces/my-workspace`)\\n3. Add remote origin\\n4. Push initial commit ### Clone Existing Workspace ```bash\\n# Clone from Gitea\\nprovisioning workspace clone workspaces/my-workspace ./workspace_my-workspace # Or using full identifier\\nprovisioning workspace clone my-workspace ./workspace_my-workspace\\n```plaintext ### Push/Pull Changes ```bash\\n# Push workspace changes\\ncd workspace_my-workspace\\nprovisioning workspace push --message \\"Updated infrastructure configs\\" # Pull latest changes\\nprovisioning workspace pull # Sync (pull + push)\\nprovisioning workspace sync\\n```plaintext ### Branch Management ```bash\\n# Create branch\\nprovisioning workspace branch create feature-new-cluster # Switch branch\\nprovisioning workspace branch switch feature-new-cluster # List branches\\nprovisioning workspace branch list # Delete branch\\nprovisioning workspace branch delete feature-new-cluster\\n```plaintext ### Git Status ```bash\\n# Get workspace git status\\nprovisioning workspace git status # Show uncommitted changes\\nprovisioning workspace git diff # Show staged changes\\nprovisioning workspace git diff --staged\\n```plaintext --- ## Workspace Locking Distributed locking prevents concurrent modifications to workspaces using Gitea issues. ### Lock Types - **read**: Multiple readers allowed, blocks writers\\n- **write**: Exclusive access, blocks all other locks\\n- **deploy**: Exclusive access for deployments ### Acquire Lock ```bash\\n# Acquire write lock\\nprovisioning gitea lock acquire my-workspace write \\\\ --operation \\"Deploying servers\\" \\\\ --expiry \\"2025-10-06T14:00:00Z\\" # Output:\\n# ✓ Lock acquired for workspace: my-workspace\\n# Lock ID: 42\\n# Type: write\\n# User: provisioning\\n```plaintext ### Check Lock Status ```bash\\n# List locks for workspace\\nprovisioning gitea lock list my-workspace # List all active locks\\nprovisioning gitea lock list # Get lock details\\nprovisioning gitea lock info my-workspace 42\\n```plaintext ### Release Lock ```bash\\n# Release lock\\nprovisioning gitea lock release my-workspace 42\\n```plaintext ### Force Release Lock (Admin) ```bash\\n# Force release stuck lock\\nprovisioning gitea lock force-release my-workspace 42 \\\\ --reason \\"Deployment failed, releasing lock\\"\\n```plaintext ### Automatic Locking Use `with-workspace-lock` for automatic lock management: ```nushell\\nuse lib_provisioning/gitea/locking.nu * with-workspace-lock \\"my-workspace\\" \\"deploy\\" \\"Server deployment\\" { # Your deployment code here # Lock automatically released on completion or error\\n}\\n```plaintext ### Lock Cleanup ```bash\\n# Cleanup expired locks\\nprovisioning gitea lock cleanup\\n```plaintext --- ## Extension Publishing Publish taskservs, providers, and clusters as versioned releases on Gitea. ### Publish Extension ```bash\\n# Publish taskserv\\nprovisioning gitea extension publish \\\\ ./extensions/taskservs/database/postgres \\\\ 1.2.0 \\\\ --release-notes \\"Added connection pooling support\\" # Publish provider\\nprovisioning gitea extension publish \\\\ ./extensions/providers/aws_prov \\\\ 2.0.0 \\\\ --prerelease # Publish cluster\\nprovisioning gitea extension publish \\\\ ./extensions/clusters/buildkit \\\\ 1.0.0\\n```plaintext This will: 1. Validate extension structure\\n2. Create git tag (if workspace is git repo)\\n3. Package extension as `.tar.gz`\\n4. Create Gitea release\\n5. Upload package as release asset ### List Published Extensions ```bash\\n# List all extensions\\nprovisioning gitea extension list # Filter by type\\nprovisioning gitea extension list --type taskserv\\nprovisioning gitea extension list --type provider\\nprovisioning gitea extension list --type cluster\\n```plaintext ### Download Extension ```bash\\n# Download specific version\\nprovisioning gitea extension download postgres 1.2.0 \\\\ --destination ./extensions/taskservs/database # Extension is downloaded and extracted automatically\\n```plaintext ### Extension Metadata ```bash\\n# Get extension information\\nprovisioning gitea extension info postgres 1.2.0\\n```plaintext ### Publishing Workflow ```bash\\n# 1. Make changes to extension\\ncd extensions/taskservs/database/postgres # 2. Update version in kcl/kcl.mod\\n# 3. Update CHANGELOG.md # 4. Commit changes\\ngit add .\\ngit commit -m \\"Release v1.2.0\\" # 5. Publish to Gitea\\nprovisioning gitea extension publish . 1.2.0\\n```plaintext --- ## Service Management ### Start/Stop Gitea ```bash\\n# Start Gitea (local mode)\\nprovisioning gitea start # Stop Gitea\\nprovisioning gitea stop # Restart Gitea\\nprovisioning gitea restart\\n```plaintext ### Check Status ```bash\\n# Get service status\\nprovisioning gitea status # Output:\\n# Gitea Status:\\n# Mode: local\\n# Deployment: docker\\n# Running: true\\n# Port: 3000\\n# URL: http://localhost:3000\\n# Container: provisioning-gitea\\n# Health: ✓ OK\\n```plaintext ### View Logs ```bash\\n# View recent logs\\nprovisioning gitea logs # Follow logs\\nprovisioning gitea logs --follow # Show specific number of lines\\nprovisioning gitea logs --lines 200\\n```plaintext ### Install Gitea Binary ```bash\\n# Install latest version\\nprovisioning gitea install # Install specific version\\nprovisioning gitea install 1.21.0 # Custom install directory\\nprovisioning gitea install --install-dir ~/bin\\n```plaintext --- ## API Reference ### Repository Operations ```nushell\\nuse lib_provisioning/gitea/api_client.nu * # Create repository\\ncreate-repository \\"my-org\\" \\"my-repo\\" \\"Description\\" true # Get repository\\nget-repository \\"my-org\\" \\"my-repo\\" # Delete repository\\ndelete-repository \\"my-org\\" \\"my-repo\\" --force # List repositories\\nlist-repositories \\"my-org\\"\\n```plaintext ### Release Operations ```nushell\\n# Create release\\ncreate-release \\"my-org\\" \\"my-repo\\" \\"v1.0.0\\" \\"Release Name\\" \\"Notes\\" # Upload asset\\nupload-release-asset \\"my-org\\" \\"my-repo\\" 123 \\"./file.tar.gz\\" # Get release\\nget-release-by-tag \\"my-org\\" \\"my-repo\\" \\"v1.0.0\\" # List releases\\nlist-releases \\"my-org\\" \\"my-repo\\"\\n```plaintext ### Workspace Operations ```nushell\\nuse lib_provisioning/gitea/workspace_git.nu * # Initialize workspace git\\ninit-workspace-git \\"./workspace_test\\" \\"test\\" --remote \\"gitea\\" # Clone workspace\\nclone-workspace \\"workspaces/my-workspace\\" \\"./workspace_my-workspace\\" # Push changes\\npush-workspace \\"./workspace_my-workspace\\" \\"Updated configs\\" # Pull changes\\npull-workspace \\"./workspace_my-workspace\\"\\n```plaintext ### Locking Operations ```nushell\\nuse lib_provisioning/gitea/locking.nu * # Acquire lock\\nlet lock = acquire-workspace-lock \\"my-workspace\\" \\"write\\" \\"Deployment\\" # Release lock\\nrelease-workspace-lock \\"my-workspace\\" $lock.lock_id # Check if locked\\nis-workspace-locked \\"my-workspace\\" \\"write\\" # List locks\\nlist-workspace-locks \\"my-workspace\\"\\n```plaintext --- ## Troubleshooting ### Gitea Not Starting **Problem**: `provisioning gitea start` fails **Solutions**: ```bash\\n# Check Docker status\\ndocker ps # Check if port is in use\\nlsof -i :3000 # Check Gitea logs\\nprovisioning gitea logs # Remove old container\\ndocker rm -f provisioning-gitea\\nprovisioning gitea start\\n```plaintext ### Token Authentication Failed **Problem**: `provisioning gitea auth validate` returns false **Solutions**: ```bash\\n# Verify token file exists\\nls ~/.provisioning/secrets/gitea-token.enc # Test decryption\\nsops --decrypt ~/.provisioning/secrets/gitea-token.enc # Regenerate token in Gitea UI\\n# Save new token\\necho \\"new-token\\" | sops --encrypt /dev/stdin > ~/.provisioning/secrets/gitea-token.enc\\n```plaintext ### Cannot Push to Repository **Problem**: Git push fails with authentication error **Solutions**: ```bash\\n# Check remote URL\\ncd workspace_my-workspace\\ngit remote -v # Reconfigure remote with token\\ngit remote set-url origin http://username:token@localhost:3000/org/repo.git # Or use SSH\\ngit remote set-url origin git@localhost:workspaces/my-workspace.git\\n```plaintext ### Lock Already Exists **Problem**: Cannot acquire lock, workspace already locked **Solutions**: ```bash\\n# Check active locks\\nprovisioning gitea lock list my-workspace # Get lock details\\nprovisioning gitea lock info my-workspace 42 # If lock is stale, force release\\nprovisioning gitea lock force-release my-workspace 42 --reason \\"Stale lock\\"\\n```plaintext ### Extension Validation Failed **Problem**: Extension publishing fails validation **Solutions**: ```bash\\n# Check extension structure\\nls -la extensions/taskservs/myservice/\\n# Required:\\n# - kcl/kcl.mod\\n# - kcl/*.k (main schema file) # Verify kcl.mod format\\ncat extensions/taskservs/myservice/kcl/kcl.mod # Should have:\\n# [package]\\n# name = \\"myservice\\"\\n# version = \\"1.0.0\\"\\n```plaintext ### Docker Volume Permissions **Problem**: Gitea Docker container has permission errors **Solutions**: ```bash\\n# Fix data directory permissions\\nsudo chown -R 1000:1000 ~/.provisioning/gitea # Or recreate with correct permissions\\nprovisioning gitea stop --remove\\nrm -rf ~/.provisioning/gitea\\nprovisioning gitea start\\n```plaintext --- ## Best Practices ### Workspace Management 1. **Always use locking** for concurrent operations\\n2. **Commit frequently** with descriptive messages\\n3. **Use branches** for experimental changes\\n4. **Sync before operations** to get latest changes ### Extension Publishing 1. **Follow semantic versioning** (MAJOR.MINOR.PATCH)\\n2. **Update CHANGELOG.md** for each release\\n3. **Test extensions** before publishing\\n4. **Use prerelease flag** for beta versions ### Security 1. **Encrypt tokens** with SOPS\\n2. **Use private repositories** for sensitive workspaces\\n3. **Rotate tokens** regularly\\n4. **Audit lock history** via Gitea issues ### Performance 1. **Cleanup expired locks** periodically\\n2. **Use shallow clones** for large workspaces\\n3. **Archive old releases** to reduce storage\\n4. **Monitor Gitea resources** for local deployments --- ## Advanced Usage ### Custom Gitea Deployment Edit `docker-compose.yml`: ```yaml\\nservices: gitea: image: gitea/gitea:1.21 environment: - GITEA__server__DOMAIN=gitea.example.com - GITEA__server__ROOT_URL=https://gitea.example.com # Add custom settings volumes: - /custom/path/gitea:/data\\n```plaintext ### Webhooks Integration Configure webhooks for automated workflows: ```kcl\\nimport provisioning.gitea as gitea _webhook = gitea.GiteaWebhook { url = \\"https://provisioning.example.com/api/webhooks/gitea\\" events = [\\"push\\", \\"pull_request\\", \\"release\\"] secret = \\"webhook-secret\\"\\n}\\n```plaintext ### Batch Extension Publishing ```bash\\n# Publish all taskservs with same version\\nprovisioning gitea extension publish-batch \\\\ ./extensions/taskservs \\\\ 1.0.0 \\\\ --extension-type taskserv\\n```plaintext --- ## References - **Gitea API Documentation**: \\n- **KCL Schema**: `/Users/Akasha/project-provisioning/provisioning/kcl/gitea.k`\\n- **API Client**: `/Users/Akasha/project-provisioning/provisioning/core/nulib/lib_provisioning/gitea/api_client.nu`\\n- **Workspace Git**: `/Users/Akasha/project-provisioning/provisioning/core/nulib/lib_provisioning/gitea/workspace_git.nu`\\n- **Locking**: `/Users/Akasha/project-provisioning/provisioning/core/nulib/lib_provisioning/gitea/locking.nu` --- **Version:** 1.0.0\\n**Maintained By:** Provisioning Team\\n**Last Updated:** 2025-10-06","breadcrumbs":"Gitea Integration Guide » Architecture","id":"1671","title":"Architecture"},"1672":{"body":"","breadcrumbs":"Service Mesh Ingress Guide » Service Mesh & Ingress Guide","id":"1672","title":"Service Mesh & Ingress Guide"},"1673":{"body":"This guide helps you choose between different service mesh and ingress controller options for your Kubernetes deployments.","breadcrumbs":"Service Mesh Ingress Guide » Comparison","id":"1673","title":"Comparison"},"1674":{"body":"Service Mesh Handles East-West traffic (service-to-service communication): Automatic mTLS encryption between services Traffic management and routing Observability and monitoring Service discovery Fault tolerance and resilience Ingress Controller Handles North-South traffic (external to internal): Route external traffic into the cluster TLS/HTTPS termination Virtual hosts and path routing Load balancing Can work with or without a service mesh","breadcrumbs":"Service Mesh Ingress Guide » Understanding the Difference","id":"1674","title":"Understanding the Difference"},"1675":{"body":"Istio Version : 1.24.0 Best for : Full-featured service mesh deployments with comprehensive observability Key Features : ✅ Comprehensive feature set ✅ Built-in Istio Gateway ingress controller ✅ Advanced traffic management ✅ Excellent observability (Kiali, Grafana, Jaeger) ✅ Virtual services, destination rules, traffic policies ✅ Mutual TLS (mTLS) with automatic certificate rotation ✅ Canary deployments and traffic mirroring Resource Requirements : CPU: 500m (Pilot) + 100m per gateway Memory: 2048Mi (Pilot) + 128Mi per gateway Relatively high overhead Pros : Industry-standard solution with large community Rich feature set for complex requirements Built-in ingress gateway (don\'t need external ingress) Strong observability capabilities Enterprise support available Cons : Significant resource overhead Complex configuration learning curve Can be overkill for simple applications Sidecar injection required for all services Use when : You need comprehensive traffic management Complex microservice patterns (canary deployments, traffic mirroring) Enterprise requirements You already understand service meshes Your team has Istio expertise Installation : provisioning taskserv create istio\\n```plaintext --- #### Linkerd **Version**: 2.16.0 **Best for**: Lightweight, high-performance service mesh with minimal complexity **Key Features**: - ✅ Ultra-lightweight (minimal resource footprint)\\n- ✅ Simple configuration\\n- ✅ Automatic mTLS with certificate rotation\\n- ✅ Fast sidecar startup (built in Rust)\\n- ✅ Live traffic visualization\\n- ✅ Service topology and dependency discovery\\n- ✅ Golden metrics out of the box (latency, success rate, throughput) **Resource Requirements**: - CPU proxy: 100m request, 1000m limit\\n- Memory proxy: 20Mi request, 250Mi limit\\n- Very lightweight compared to Istio **Pros**: - Minimal resource overhead\\n- Simple, intuitive configuration\\n- Fast startup and deployment\\n- Built in Rust for performance\\n- Excellent golden metrics\\n- Good for resource-constrained environments\\n- Can run alongside Istio **Cons**: - Fewer advanced features than Istio\\n- Requires external ingress controller\\n- Smaller ecosystem and fewer integrations\\n- Less feature-rich traffic management\\n- Requires cert-manager for mTLS **Use when**: - You want simplicity and minimal overhead\\n- Running on resource-constrained clusters\\n- You prefer straightforward configuration\\n- You don\'t need advanced traffic management\\n- You\'re using Kubernetes 1.21+ **Installation**: ```bash\\n# Linkerd requires cert-manager\\nprovisioning taskserv create cert-manager\\nprovisioning taskserv create linkerd\\nprovisioning taskserv create nginx-ingress # Or traefik/contour\\n```plaintext --- #### Cilium **Version**: See existing Cilium taskserv **Best for**: CNI-based networking with integrated service mesh **Key Features**: - ✅ CNI and service mesh in one solution\\n- ✅ eBPF-based for high performance\\n- ✅ Network policy enforcement\\n- ✅ Service mesh mode (optional)\\n- ✅ Hubble for observability\\n- ✅ Cluster mesh for multi-cluster **Pros**: - Replaces CNI plugin entirely\\n- High-performance eBPF kernel networking\\n- Can serve as both CNI and service mesh\\n- No sidecar needed (uses eBPF)\\n- Network policy support **Cons**: - Requires Linux kernel with eBPF support\\n- Service mesh mode is secondary feature\\n- More complex than Linkerd\\n- Not as mature in service mesh role **Use when**: - You need both CNI and service mesh\\n- You\'re on modern Linux kernels with eBPF\\n- You want kernel-level networking --- ### Ingress Controller Options #### Nginx Ingress **Version**: 1.12.0 **Best for**: Most Kubernetes deployments - proven, reliable, widely supported **Key Features**: - ✅ Battle-tested and production-proven\\n- ✅ Most popular ingress controller\\n- ✅ Extensive documentation and community\\n- ✅ Rich configuration options\\n- ✅ SSL/TLS termination\\n- ✅ URL rewriting and routing\\n- ✅ Rate limiting and DDoS protection **Pros**: - Proven stability in production\\n- Widest community and ecosystem\\n- Extensive documentation\\n- Multiple commercial support options\\n- Works with any service mesh\\n- Moderate resource footprint **Cons**: - Configuration can be verbose\\n- Limited middleware ecosystem (compared to Traefik)\\n- No automatic TLS with Let\'s Encrypt\\n- Configuration via annotations **Use when**: - You want proven stability\\n- Wide community support is important\\n- You need traditional ingress controller\\n- You\'re building production systems\\n- You want abundant documentation **Installation**: ```bash\\nprovisioning taskserv create nginx-ingress\\n```plaintext **With Linkerd**: ```bash\\nprovisioning taskserv create linkerd\\nprovisioning taskserv create nginx-ingress\\n```plaintext --- #### Traefik **Version**: 3.3.0 **Best for**: Modern cloud-native applications with dynamic service discovery **Key Features**: - ✅ Automatic service discovery\\n- ✅ Native Let\'s Encrypt support\\n- ✅ Middleware system for advanced routing\\n- ✅ Built-in dashboard and metrics\\n- ✅ API-driven configuration\\n- ✅ Dynamic configuration updates\\n- ✅ Support for multiple protocols (HTTP, TCP, gRPC) **Pros**: - Modern, cloud-native design\\n- Automatic TLS with Let\'s Encrypt\\n- Middleware ecosystem for extensibility\\n- Built-in dashboard for monitoring\\n- Dynamic configuration without restart\\n- API-driven approach\\n- Growing community **Cons**: - Different configuration paradigm (IngressRoute CRD)\\n- Smaller community than Nginx\\n- Learning curve for traditional ops\\n- Less mature than Nginx **Use when**: - You want modern cloud-native features\\n- Automatic TLS is important\\n- You like middleware-based routing\\n- You want dynamic configuration\\n- You\'re building microservices platforms **Installation**: ```bash\\nprovisioning taskserv create traefik\\n```plaintext **With Linkerd**: ```bash\\nprovisioning taskserv create linkerd\\nprovisioning taskserv create traefik\\n```plaintext --- #### Contour **Version**: 1.31.0 **Best for**: Envoy-based ingress with simple CRD configuration **Key Features**: - ✅ Envoy proxy backend (same as Istio)\\n- ✅ Simple CRD-based configuration\\n- ✅ HTTPProxy CRD for advanced routing\\n- ✅ Service delegation and composition\\n- ✅ External authorization\\n- ✅ Rate limiting support **Pros**: - Uses same Envoy proxy as Istio\\n- Simple but powerful configuration\\n- Good for multi-tenant clusters\\n- CRD-based (declarative)\\n- Good documentation **Cons**: - Smaller community than Nginx/Traefik\\n- Fewer integrations and plugins\\n- Less feature-rich than Traefik\\n- Fewer real-world examples **Use when**: - You want Envoy proxy for consistency with Istio\\n- You prefer simple configuration\\n- You like CRD-based approach\\n- You need multi-tenant support **Installation**: ```bash\\nprovisioning taskserv create contour\\n```plaintext --- #### HAProxy Ingress **Version**: 0.15.0 **Best for**: High-performance environments requiring advanced load balancing **Key Features**: - ✅ HAProxy backend for performance\\n- ✅ Advanced load balancing algorithms\\n- ✅ High throughput\\n- ✅ Flexible configuration\\n- ✅ Proven performance **Pros**: - Excellent performance\\n- Advanced load balancing options\\n- Battle-tested HAProxy backend\\n- Good for high-traffic scenarios **Cons**: - Less Kubernetes-native than others\\n- Smaller community\\n- Configuration complexity\\n- Fewer modern features **Use when**: - Performance is critical\\n- High traffic is expected\\n- You need advanced load balancing --- ## Recommended Combinations ### 1. Linkerd + Nginx Ingress (Recommended for most users) **Why**: Lightweight mesh + proven ingress = great balance ```bash\\nprovisioning taskserv create cert-manager\\nprovisioning taskserv create linkerd\\nprovisioning taskserv create nginx-ingress\\n```plaintext **Pros**: - Minimal overhead\\n- Simple to manage\\n- Proven stability\\n- Good observability **Cons**: - Less advanced features than Istio --- ### 2. Istio (Standalone) **Why**: All-in-one service mesh with built-in gateway ```bash\\nprovisioning taskserv create istio\\n```plaintext **Pros**: - Unified traffic management\\n- Powerful observability\\n- No external ingress needed\\n- Rich features **Cons**: - Higher resource usage\\n- More complex --- ### 3. Linkerd + Traefik **Why**: Lightweight mesh + modern ingress ```bash\\nprovisioning taskserv create cert-manager\\nprovisioning taskserv create linkerd\\nprovisioning taskserv create traefik\\n```plaintext **Pros**: - Minimal overhead\\n- Modern features\\n- Automatic TLS --- ### 4. No Mesh + Nginx Ingress (Simple deployments) **Why**: Just get traffic in without service mesh ```bash\\nprovisioning taskserv create nginx-ingress\\n```plaintext **Pros**: - Simplest setup\\n- Minimal overhead\\n- Proven stability --- ## Decision Matrix | Requirement | Istio | Linkerd | Cilium | Nginx | Traefik | Contour | HAProxy |\\n|-----------|-------|---------|--------|-------|---------|---------|---------|\\n| Lightweight | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |\\n| Simple Config | ❌ | ✅ | ⚠️ | ⚠️ | ✅ | ✅ | ❌ |\\n| Full Features | ✅ | ⚠️ | ✅ | ⚠️ | ✅ | ⚠️ | ✅ |\\n| Auto TLS | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |\\n| Service Mesh | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |\\n| Performance | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |\\n| Community | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ | ⚠️ | ## Migration Paths ### From Istio to Linkerd 1. Install Linkerd alongside Istio\\n2. Gradually migrate services (add Linkerd annotations)\\n3. Verify Linkerd handles traffic correctly\\n4. Install external ingress controller (Nginx/Traefik)\\n5. Update Istio Virtual Services to use new ingress\\n6. Remove Istio once migration complete ### Between Ingress Controllers 1. Install new ingress controller\\n2. Create duplicate Ingress resources pointing to new controller\\n3. Test with new ingress (use IngressClassName)\\n4. Update DNS/load balancer to point to new ingress\\n5. Drain connections from old ingress\\n6. Remove old ingress controller --- ## Examples Complete examples of how to configure service meshes and ingress controllers in your workspace. ### Example 1: Linkerd + Nginx Ingress Deployment This is the recommended configuration for most deployments - lightweight and proven. #### Step 1: Create Taskserv Configurations **File**: `workspace/infra/my-cluster/taskservs/cert-manager.k` ```kcl\\nimport provisioning.extensions.taskservs.infrastructure.cert_manager as cm # Cert-manager is required for Linkerd\'s mTLS certificates\\n_taskserv = cm.CertManager { version = \\"v1.15.0\\" namespace = \\"cert-manager\\"\\n}\\n```plaintext **File**: `workspace/infra/my-cluster/taskservs/linkerd.k` ```kcl\\nimport provisioning.extensions.taskservs.networking.linkerd as linkerd # Lightweight service mesh with minimal overhead\\n_taskserv = linkerd.Linkerd { version = \\"2.16.0\\" namespace = \\"linkerd\\" # Enable observability ha_mode = False # Use True for production HA viz_enabled = True prometheus = True grafana = True # Use cert-manager for mTLS certificates cert_manager = True trust_domain = \\"cluster.local\\" # Resource configuration (very lightweight) resources = { proxy_cpu_request = \\"100m\\" proxy_cpu_limit = \\"1000m\\" proxy_memory_request = \\"20Mi\\" proxy_memory_limit = \\"250Mi\\" }\\n}\\n```plaintext **File**: `workspace/infra/my-cluster/taskservs/nginx-ingress.k` ```kcl\\nimport provisioning.extensions.taskservs.networking.nginx_ingress as nginx # Battle-tested ingress controller\\n_taskserv = nginx.NginxIngress { version = \\"1.12.0\\" namespace = \\"ingress-nginx\\" # Deployment configuration deployment_type = \\"Deployment\\" # Or \\"DaemonSet\\" for node-local ingress replicas = 2 # Enable metrics for observability prometheus_metrics = True # Resource allocation resources = { cpu_request = \\"100m\\" cpu_limit = \\"1000m\\" memory_request = \\"90Mi\\" memory_limit = \\"500Mi\\" }\\n}\\n```plaintext #### Step 2: Deploy Service Mesh Components ```bash\\n# Install cert-manager (prerequisite for Linkerd)\\nprovisioning taskserv create cert-manager # Install Linkerd service mesh\\nprovisioning taskserv create linkerd # Install Nginx ingress controller\\nprovisioning taskserv create nginx-ingress # Verify installation\\nlinkerd check\\nkubectl get deploy -n ingress-nginx\\n```plaintext #### Step 3: Configure Application Deployment **File**: `workspace/infra/my-cluster/clusters/web-api.k` ```kcl\\nimport provisioning.kcl.k8s_deploy as k8s\\nimport provisioning.extensions.taskservs.networking.nginx_ingress as nginx # Define the web API service with Linkerd service mesh and Nginx ingress\\nservice = k8s.K8sDeploy { # Basic information name = \\"web-api\\" namespace = \\"production\\" create_ns = True # Service mesh configuration - use Linkerd service_mesh = \\"linkerd\\" service_mesh_ns = \\"linkerd\\" service_mesh_config = { mtls_enabled = True tracing_enabled = False } # Ingress configuration - use Nginx ingress_controller = \\"nginx\\" ingress_ns = \\"ingress-nginx\\" ingress_config = { tls_enabled = True default_backend = \\"web-api:8080\\" } # Deployment spec spec = { replicas = 3 containers = [ { name = \\"api\\" image = \\"myregistry.azurecr.io/web-api:v1.0.0\\" imagePull = \\"Always\\" ports = [ { name = \\"http\\" typ = \\"TCP\\" container = 8080 } ] } ] } # Kubernetes service service = { name = \\"web-api\\" typ = \\"ClusterIP\\" ports = [ { name = \\"http\\" typ = \\"TCP\\" target = 8080 } ] }\\n}\\n```plaintext #### Step 4: Create Ingress Resource **File**: `workspace/infra/my-cluster/ingress/web-api-ingress.yaml` ```yaml\\napiVersion: networking.k8s.io/v1\\nkind: Ingress\\nmetadata: name: web-api namespace: production annotations: cert-manager.io/cluster-issuer: letsencrypt-prod nginx.ingress.kubernetes.io/rewrite-target: /\\nspec: ingressClassName: nginx tls: - hosts: - api.example.com secretName: web-api-tls rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: web-api port: number: 8080\\n```plaintext --- ### Example 2: Istio (Standalone) Deployment Complete service mesh with built-in ingress gateway. #### Step 1: Install Istio **File**: `workspace/infra/my-cluster/taskservs/istio.k` ```kcl\\nimport provisioning.extensions.taskservs.networking.istio as istio # Full-featured service mesh\\n_taskserv = istio.Istio { version = \\"1.24.0\\" profile = \\"default\\" # Options: default, demo, minimal, remote namespace = \\"istio-system\\" # Core features mtls_enabled = True mtls_mode = \\"PERMISSIVE\\" # Start with PERMISSIVE, switch to STRICT when ready # Traffic management ingress_gateway = True egress_gateway = False # Observability tracing = { enabled = True provider = \\"jaeger\\" sampling_rate = 0.1 # Sample 10% for production } prometheus = True grafana = True kiali = True # Resource configuration resources = { pilot_cpu = \\"500m\\" pilot_memory = \\"2048Mi\\" gateway_cpu = \\"100m\\" gateway_memory = \\"128Mi\\" }\\n}\\n```plaintext #### Step 2: Deploy Istio ```bash\\n# Install Istio\\nprovisioning taskserv create istio # Verify installation\\nistioctl verify-install\\n```plaintext #### Step 3: Configure Application with Istio **File**: `workspace/infra/my-cluster/clusters/api-service.k` ```kcl\\nimport provisioning.kcl.k8s_deploy as k8s service = k8s.K8sDeploy { name = \\"api-service\\" namespace = \\"production\\" create_ns = True # Use Istio for both service mesh AND ingress service_mesh = \\"istio\\" service_mesh_ns = \\"istio-system\\" ingress_controller = \\"istio-gateway\\" # Istio\'s built-in gateway spec = { replicas = 3 containers = [ { name = \\"api\\" image = \\"myregistry.azurecr.io/api:v1.0.0\\" ports = [ { name = \\"http\\", typ = \\"TCP\\", container = 8080 } ] } ] } service = { name = \\"api-service\\" typ = \\"ClusterIP\\" ports = [ { name = \\"http\\", typ = \\"TCP\\", target = 8080 } ] } # Istio-specific proxy configuration prxyGatewayServers = [ { port = { number = 80, protocol = \\"HTTP\\", name = \\"http\\" } hosts = [\\"api.example.com\\"] }, { port = { number = 443, protocol = \\"HTTPS\\", name = \\"https\\" } hosts = [\\"api.example.com\\"] tls = { mode = \\"SIMPLE\\" credentialName = \\"api-tls-cert\\" } } ] # Virtual service routing configuration prxyVirtualService = { hosts = [\\"api.example.com\\"] gateways = [\\"api-gateway\\"] matches = [ { typ = \\"http\\" location = [ { port = 80 } ] route_destination = [ { port_number = 8080, host = \\"api-service\\" } ] } ] }\\n}\\n```plaintext --- ### Example 3: Linkerd + Traefik (Modern Cloud-Native) Lightweight mesh with modern ingress controller and automatic TLS. #### Step 1: Create Configurations **File**: `workspace/infra/my-cluster/taskservs/linkerd.k` ```kcl\\nimport provisioning.extensions.taskservs.networking.linkerd as linkerd _taskserv = linkerd.Linkerd { version = \\"2.16.0\\" namespace = \\"linkerd\\" viz_enabled = True prometheus = True\\n}\\n```plaintext **File**: `workspace/infra/my-cluster/taskservs/traefik.k` ```kcl\\nimport provisioning.extensions.taskservs.networking.traefik as traefik # Modern ingress with middleware and auto-TLS\\n_taskserv = traefik.Traefik { version = \\"3.3.0\\" namespace = \\"traefik\\" replicas = 2 dashboard = True metrics = True access_logs = True # Enable Let\'s Encrypt for automatic TLS lets_encrypt = True lets_encrypt_email = \\"admin@example.com\\" resources = { cpu_request = \\"100m\\" cpu_limit = \\"1000m\\" memory_request = \\"128Mi\\" memory_limit = \\"512Mi\\" }\\n}\\n```plaintext #### Step 2: Deploy ```bash\\nprovisioning taskserv create cert-manager\\nprovisioning taskserv create linkerd\\nprovisioning taskserv create traefik\\n```plaintext #### Step 3: Create Traefik IngressRoute **File**: `workspace/infra/my-cluster/ingress/api-route.yaml` ```yaml\\napiVersion: traefik.io/v1alpha1\\nkind: IngressRoute\\nmetadata: name: api namespace: production\\nspec: entryPoints: - websecure routes: - match: Host(`api.example.com`) kind: Rule services: - name: api-service port: 8080 tls: certResolver: letsencrypt domains: - main: api.example.com\\n```plaintext --- ### Example 4: Minimal Setup (Just Nginx, No Service Mesh) For simple deployments that don\'t need service mesh. #### Step 1: Install Nginx **File**: `workspace/infra/my-cluster/taskservs/nginx-ingress.k` ```kcl\\nimport provisioning.extensions.taskservs.networking.nginx_ingress as nginx _taskserv = nginx.NginxIngress { version = \\"1.12.0\\" replicas = 2 prometheus_metrics = True\\n}\\n```plaintext #### Step 2: Deploy ```bash\\nprovisioning taskserv create nginx-ingress\\n```plaintext #### Step 3: Application Configuration **File**: `workspace/infra/my-cluster/clusters/simple-app.k` ```kcl\\nimport provisioning.kcl.k8s_deploy as k8s service = k8s.K8sDeploy { name = \\"simple-app\\" namespace = \\"default\\" # No service mesh - just ingress ingress_controller = \\"nginx\\" ingress_ns = \\"ingress-nginx\\" spec = { replicas = 2 containers = [ { name = \\"app\\" image = \\"nginx:latest\\" ports = [{ name = \\"http\\", typ = \\"TCP\\", container = 80 }] } ] } service = { name = \\"simple-app\\" typ = \\"ClusterIP\\" ports = [{ name = \\"http\\", typ = \\"TCP\\", target = 80 }] }\\n}\\n```plaintext #### Step 4: Create Ingress **File**: `workspace/infra/my-cluster/ingress/simple-app-ingress.yaml` ```yaml\\napiVersion: networking.k8s.io/v1\\nkind: Ingress\\nmetadata: name: simple-app namespace: default\\nspec: ingressClassName: nginx rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: simple-app port: number: 80\\n```plaintext --- ## Enable Sidecar Injection for Services ### For Linkerd ```bash\\n# Label namespace for automatic sidecar injection\\nkubectl annotate namespace production linkerd.io/inject=enabled # Or add annotation to specific deployment\\nkubectl annotate pod my-pod linkerd.io/inject=enabled\\n```plaintext ### For Istio ```bash\\n# Label namespace for automatic sidecar injection\\nkubectl label namespace production istio-injection=enabled # Verify injection\\nkubectl describe pod -n production | grep istio-proxy\\n```plaintext --- ## Monitoring and Observability ### Linkerd Dashboard ```bash\\n# Open Linkerd Viz dashboard\\nlinkerd viz dashboard # View service topology\\nlinkerd viz stat ns\\nlinkerd viz tap -n production\\n```plaintext ### Istio Dashboards ```bash\\n# Kiali (service mesh visualization)\\nkubectl port-forward -n istio-system svc/kiali 20000:20000\\n# http://localhost:20000 # Grafana (metrics)\\nkubectl port-forward -n istio-system svc/grafana 3000:3000\\n# http://localhost:3000 (default: admin/admin) # Jaeger (distributed tracing)\\nkubectl port-forward -n istio-system svc/jaeger-query 16686:16686\\n# http://localhost:16686\\n```plaintext ### Traefik Dashboard ```bash\\n# Forward Traefik dashboard\\nkubectl port-forward -n traefik svc/traefik 8080:8080\\n# http://localhost:8080/dashboard/\\n```plaintext --- ## Quick Reference ### Installation Commands #### Service Mesh - Istio ```bash\\n# Install Istio (includes built-in ingress gateway)\\nprovisioning taskserv create istio # Verify installation\\nistioctl verify-install # Enable sidecar injection on namespace\\nkubectl label namespace default istio-injection=enabled # View Kiali dashboard\\nkubectl port-forward -n istio-system svc/kiali 20000:20000\\n# Open: http://localhost:20000\\n```plaintext #### Service Mesh - Linkerd ```bash\\n# Install cert-manager first (Linkerd requirement)\\nprovisioning taskserv create cert-manager # Install Linkerd\\nprovisioning taskserv create linkerd # Verify installation\\nlinkerd check # Enable automatic sidecar injection\\nkubectl annotate namespace default linkerd.io/inject=enabled # View live dashboard\\nlinkerd viz dashboard\\n```plaintext #### Ingress Controllers ```bash\\n# Install Nginx Ingress (most popular)\\nprovisioning taskserv create nginx-ingress # Install Traefik (modern cloud-native)\\nprovisioning taskserv create traefik # Install Contour (Envoy-based)\\nprovisioning taskserv create contour # Install HAProxy Ingress (high-performance)\\nprovisioning taskserv create haproxy-ingress\\n```plaintext ### Common Installation Combinations #### Option 1: Linkerd + Nginx Ingress (Recommended) **Lightweight mesh + proven ingress** ```bash\\n# Step 1: Install cert-manager\\nprovisioning taskserv create cert-manager # Step 2: Install Linkerd\\nprovisioning taskserv create linkerd # Step 3: Install Nginx Ingress\\nprovisioning taskserv create nginx-ingress # Step 4: Verify installation\\nlinkerd check\\nkubectl get deploy -n ingress-nginx # Step 5: Create sample application with Linkerd\\nkubectl annotate namespace default linkerd.io/inject=enabled\\nkubectl apply -f my-app.yaml\\n```plaintext #### Option 2: Istio (Standalone) **Full-featured service mesh with built-in gateway** ```bash\\n# Install Istio\\nprovisioning taskserv create istio # Verify\\nistioctl verify-install # Enable sidecar injection\\nkubectl label namespace default istio-injection=enabled # Deploy applications\\nkubectl apply -f my-app.yaml\\n```plaintext #### Option 3: Linkerd + Traefik **Lightweight mesh + modern ingress with auto TLS** ```bash\\n# Install prerequisites\\nprovisioning taskserv create cert-manager # Install service mesh\\nprovisioning taskserv create linkerd # Install modern ingress with Let\'s Encrypt\\nprovisioning taskserv create traefik # Enable sidecar injection\\nkubectl annotate namespace default linkerd.io/inject=enabled\\n```plaintext #### Option 4: Just Nginx Ingress (No Mesh) **Simple deployments without service mesh** ```bash\\n# Install ingress controller\\nprovisioning taskserv create nginx-ingress # Deploy applications\\nkubectl apply -f ingress.yaml\\n```plaintext ### Verification Commands #### Check Linkerd ```bash\\n# Full system check\\nlinkerd check # Specific component checks\\nlinkerd check --pre # Pre-install checks\\nlinkerd check -n linkerd # Linkerd namespace\\nlinkerd check -n default # Custom namespace # View version\\nlinkerd version --client\\nlinkerd version --server\\n```plaintext #### Check Istio ```bash\\n# Full system analysis\\nistioctl analyze # By namespace\\nistioctl analyze -n default # Verify configuration\\nistioctl verify-install # Check version\\nistioctl version\\n```plaintext #### Check Ingress Controllers ```bash\\n# List ingress resources\\nkubectl get ingress -A # Get ingress details\\nkubectl describe ingress -n default # Nginx specific\\nkubectl get deploy -n ingress-nginx\\nkubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx # Traefik specific\\nkubectl get deploy -n traefik\\nkubectl logs -n traefik deployment/traefik\\n```plaintext ### Troubleshooting #### Service Mesh Issues ```bash\\n# Linkerd - Check proxy status\\nlinkerd check -n  # Linkerd - View service topology\\nlinkerd tap -n  deployment/ # Istio - Check sidecar injection\\nkubectl describe pod -n  # Look for istio-proxy container # Istio - View traffic policies\\nistioctl analyze\\n```plaintext #### Ingress Controller Issues ```bash\\n# Check ingress controller logs\\nkubectl logs -n ingress-nginx deployment/ingress-nginx-controller\\nkubectl logs -n traefik deployment/traefik # Describe ingress resource\\nkubectl describe ingress  -n  # Check ingress controller service\\nkubectl get svc -n ingress-nginx\\nkubectl get svc -n traefik\\n```plaintext ### Uninstallation #### Remove Linkerd ```bash\\n# Remove annotations from namespaces\\nkubectl annotate namespace  linkerd.io/inject- --all # Uninstall Linkerd\\nlinkerd uninstall | kubectl delete -f - # Remove Linkerd namespace\\nkubectl delete namespace linkerd\\n```plaintext #### Remove Istio ```bash\\n# Remove labels from namespaces\\nkubectl label namespace  istio-injection- --all # Uninstall Istio\\nistioctl uninstall --purge # Remove Istio namespace\\nkubectl delete namespace istio-system\\n```plaintext #### Remove Ingress Controllers ```bash\\n# Nginx\\nhelm uninstall ingress-nginx -n ingress-nginx\\nkubectl delete namespace ingress-nginx # Traefik\\nhelm uninstall traefik -n traefik\\nkubectl delete namespace traefik\\n```plaintext ### Performance Tuning #### Linkerd Resource Limits ```bash\\n# Adjust proxy resource limits in linkerd.k\\n_taskserv = linkerd.Linkerd { resources: { proxy_cpu_limit = \\"2000m\\" # Increase if needed proxy_memory_limit = \\"512Mi\\" # Increase if needed }\\n}\\n```plaintext #### Istio Profile Selection ```bash\\n# Different resource profiles available\\nprofile = \\"default\\" # Full features (default)\\nprofile = \\"demo\\" # Demo mode (more resources)\\nprofile = \\"minimal\\" # Minimal (lower resources)\\nprofile = \\"remote\\" # Control plane only (advanced)\\n```plaintext --- ## Complete Workspace Directory Structure After implementing these examples, your workspace should look like: ```plaintext\\nworkspace/infra/my-cluster/\\n├── taskservs/\\n│ ├── cert-manager.k # For Linkerd mTLS\\n│ ├── linkerd.k # Service mesh option\\n│ ├── istio.k # OR Istio option\\n│ ├── nginx-ingress.k # Ingress controller\\n│ └── traefik.k # Alternative ingress\\n├── clusters/\\n│ ├── web-api.k # Application with Linkerd + Nginx\\n│ ├── api-service.k # Application with Istio\\n│ └── simple-app.k # App without service mesh\\n├── ingress/\\n│ ├── web-api-ingress.yaml # Nginx Ingress resource\\n│ ├── api-route.yaml # Traefik IngressRoute\\n│ └── simple-app-ingress.yaml # Simple Ingress\\n└── config.toml # Infrastructure-specific config\\n```plaintext --- ## Next Steps 1. **Choose your deployment model** (Linkerd+Nginx, Istio, or plain Nginx)\\n2. **Create taskserv KCL files** in `workspace/infra//taskservs/`\\n3. **Install components** using `provisioning taskserv create`\\n4. **Create application deployments** with appropriate mesh/ingress configuration\\n5. **Monitor and observe** using the appropriate dashboard --- ## Additional Resources - **Linkerd Documentation**: \\n- **Istio Documentation**: \\n- **Nginx Ingress**: \\n- **Traefik Documentation**: \\n- **Contour Documentation**: \\n- **Cilium Documentation**: ","breadcrumbs":"Service Mesh Ingress Guide » Service Mesh Options","id":"1675","title":"Service Mesh Options"},"1676":{"body":"Version : 1.0.0 Date : 2025-10-06 Audience : Users and Developers","breadcrumbs":"OCI Registry Guide » OCI Registry User Guide","id":"1676","title":"OCI Registry User Guide"},"1677":{"body":"Overview Quick Start OCI Commands Reference Dependency Management Extension Development Registry Setup Troubleshooting","breadcrumbs":"OCI Registry Guide » Table of Contents","id":"1677","title":"Table of Contents"},"1678":{"body":"The OCI registry integration enables distribution and management of provisioning extensions as OCI artifacts. This provides: Standard Distribution : Use industry-standard OCI registries Version Management : Proper semantic versioning for all extensions Dependency Resolution : Automatic dependency management Caching : Efficient caching to reduce downloads Security : TLS, authentication, and vulnerability scanning support","breadcrumbs":"OCI Registry Guide » Overview","id":"1678","title":"Overview"},"1679":{"body":"OCI (Open Container Initiative) artifacts are packaged files distributed through container registries. Unlike Docker images which contain applications, OCI artifacts can contain any type of content - in our case, provisioning extensions (KCL schemas, Nushell scripts, templates, etc.).","breadcrumbs":"OCI Registry Guide » What are OCI Artifacts?","id":"1679","title":"What are OCI Artifacts?"},"168":{"body":"Check that Kubernetes is running: # List installed task services\\nprovisioning taskserv list --infra my-infra # Check Kubernetes status\\nprovisioning server ssh dev-server-01\\nkubectl get nodes # On the server\\nexit # Or remotely\\nprovisioning server exec dev-server-01 -- kubectl get nodes","breadcrumbs":"First Deployment » Step 8: Verify Installation","id":"168","title":"Step 8: Verify Installation"},"1680":{"body":"","breadcrumbs":"OCI Registry Guide » Quick Start","id":"1680","title":"Quick Start"},"1681":{"body":"Install one of the following OCI tools: # ORAS (recommended)\\nbrew install oras # Crane (Google\'s tool)\\ngo install github.com/google/go-containerregistry/cmd/crane@latest # Skopeo (RedHat\'s tool)\\nbrew install skopeo\\n```plaintext ### 1. Start Local OCI Registry (Development) ```bash\\n# Start lightweight OCI registry (Zot)\\nprovisioning oci-registry start # Verify registry is running\\ncurl http://localhost:5000/v2/_catalog\\n```plaintext ### 2. Pull an Extension ```bash\\n# Pull Kubernetes extension from registry\\nprovisioning oci pull kubernetes:1.28.0 # Pull with specific registry\\nprovisioning oci pull kubernetes:1.28.0 \\\\ --registry harbor.company.com \\\\ --namespace provisioning-extensions\\n```plaintext ### 3. List Available Extensions ```bash\\n# List all extensions\\nprovisioning oci list # Search for specific extension\\nprovisioning oci search kubernetes # Show available versions\\nprovisioning oci tags kubernetes\\n```plaintext ### 4. Configure Workspace to Use OCI Edit `workspace/config/provisioning.yaml`: ```yaml\\ndependencies: extensions: source_type: \\"oci\\" oci: registry: \\"localhost:5000\\" namespace: \\"provisioning-extensions\\" tls_enabled: false modules: taskservs: - \\"oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0\\" - \\"oci://localhost:5000/provisioning-extensions/containerd:1.7.0\\"\\n```plaintext ### 5. Resolve Dependencies ```bash\\n# Resolve and install all dependencies\\nprovisioning dep resolve # Check what will be installed\\nprovisioning dep resolve --dry-run # Show dependency tree\\nprovisioning dep tree kubernetes\\n```plaintext --- ## OCI Commands Reference ### Pull Extension **Download extension from OCI registry** ```bash\\nprovisioning oci pull : [OPTIONS] # Examples:\\nprovisioning oci pull kubernetes:1.28.0\\nprovisioning oci pull redis:7.0.0 --registry harbor.company.com\\nprovisioning oci pull postgres:15.0 --insecure # Skip TLS verification\\n```plaintext **Options**: - `--registry `: Override registry (default: from config)\\n- `--namespace `: Override namespace (default: provisioning-extensions)\\n- `--destination `: Local installation path\\n- `--insecure`: Skip TLS certificate verification --- ### Push Extension **Publish extension to OCI registry** ```bash\\nprovisioning oci push    [OPTIONS] # Examples:\\nprovisioning oci push ./extensions/taskservs/redis redis 1.0.0\\nprovisioning oci push ./my-provider aws 2.1.0 --registry localhost:5000\\n```plaintext **Options**: - `--registry `: Target registry\\n- `--namespace `: Target namespace\\n- `--insecure`: Skip TLS verification **Prerequisites**: - Extension must have valid `manifest.yaml`\\n- Must be logged in to registry (see `oci login`) --- ### List Extensions **Show available extensions in registry** ```bash\\nprovisioning oci list [OPTIONS] # Examples:\\nprovisioning oci list\\nprovisioning oci list --namespace provisioning-platform\\nprovisioning oci list --registry harbor.company.com\\n```plaintext **Output**: ```plaintext\\n┬───────────────┬──────────────────┬─────────────────────────┬─────────────────────────────────────────────┐\\n│ name │ registry │ namespace │ reference │\\n├───────────────┼──────────────────┼─────────────────────────┼─────────────────────────────────────────────┤\\n│ kubernetes │ localhost:5000 │ provisioning-extensions │ localhost:5000/provisioning-extensions/... │\\n│ containerd │ localhost:5000 │ provisioning-extensions │ localhost:5000/provisioning-extensions/... │\\n│ cilium │ localhost:5000 │ provisioning-extensions │ localhost:5000/provisioning-extensions/... │\\n└───────────────┴──────────────────┴─────────────────────────┴─────────────────────────────────────────────┘\\n```plaintext --- ### Search Extensions **Search for extensions matching query** ```bash\\nprovisioning oci search  [OPTIONS] # Examples:\\nprovisioning oci search kube\\nprovisioning oci search postgres\\nprovisioning oci search \\"container-*\\"\\n```plaintext --- ### Show Tags (Versions) **Display all available versions of an extension** ```bash\\nprovisioning oci tags  [OPTIONS] # Examples:\\nprovisioning oci tags kubernetes\\nprovisioning oci tags redis --registry harbor.company.com\\n```plaintext **Output**: ```plaintext\\n┬────────────┬─────────┬──────────────────────────────────────────────────────┐\\n│ artifact │ version │ reference │\\n├────────────┼─────────┼──────────────────────────────────────────────────────┤\\n│ kubernetes │ 1.29.0 │ localhost:5000/provisioning-extensions/kubernetes... │\\n│ kubernetes │ 1.28.0 │ localhost:5000/provisioning-extensions/kubernetes... │\\n│ kubernetes │ 1.27.0 │ localhost:5000/provisioning-extensions/kubernetes... │\\n└────────────┴─────────┴──────────────────────────────────────────────────────┘\\n```plaintext --- ### Inspect Extension **Show detailed manifest and metadata** ```bash\\nprovisioning oci inspect : [OPTIONS] # Examples:\\nprovisioning oci inspect kubernetes:1.28.0\\nprovisioning oci inspect redis:7.0.0 --format json\\n```plaintext **Output**: ```yaml\\nname: kubernetes\\ntype: taskserv\\nversion: 1.28.0\\ndescription: Kubernetes container orchestration platform\\nauthor: Provisioning Team\\nlicense: MIT\\ndependencies: containerd: \\">=1.7.0\\" etcd: \\">=3.5.0\\"\\nplatforms: - linux/amd64 - linux/arm64\\n```plaintext --- ### Login to Registry **Authenticate with OCI registry** ```bash\\nprovisioning oci login  [OPTIONS] # Examples:\\nprovisioning oci login localhost:5000\\nprovisioning oci login harbor.company.com --username admin\\nprovisioning oci login registry.io --password-stdin < token.txt\\nprovisioning oci login registry.io --token-file ~/.provisioning/tokens/registry\\n```plaintext **Options**: - `--username `: Username (default: `_token`)\\n- `--password-stdin`: Read password from stdin\\n- `--token-file `: Read token from file **Note**: Credentials are stored in Docker config (`~/.docker/config.json`) --- ### Logout from Registry **Remove stored credentials** ```bash\\nprovisioning oci logout  # Example:\\nprovisioning oci logout harbor.company.com\\n```plaintext --- ### Delete Extension **Remove extension from registry** ```bash\\nprovisioning oci delete : [OPTIONS] # Examples:\\nprovisioning oci delete kubernetes:1.27.0\\nprovisioning oci delete redis:6.0.0 --force # Skip confirmation\\n```plaintext **Options**: - `--force`: Skip confirmation prompt\\n- `--registry `: Target registry\\n- `--namespace `: Target namespace **Warning**: This operation is irreversible. Use with caution. --- ### Copy Extension **Copy extension between registries** ```bash\\nprovisioning oci copy   [OPTIONS] # Examples:\\n# Copy between namespaces in same registry\\nprovisioning oci copy \\\\ localhost:5000/test/kubernetes:1.28.0 \\\\ localhost:5000/production/kubernetes:1.28.0 # Copy between different registries\\nprovisioning oci copy \\\\ localhost:5000/provisioning-extensions/kubernetes:1.28.0 \\\\ harbor.company.com/provisioning/kubernetes:1.28.0\\n```plaintext --- ### Show OCI Configuration **Display current OCI settings** ```bash\\nprovisioning oci config # Output:\\n{ tool: \\"oras\\" registry: \\"localhost:5000\\" namespace: { extensions: \\"provisioning-extensions\\" platform: \\"provisioning-platform\\" } cache_dir: \\"~/.provisioning/oci-cache\\" tls_enabled: false\\n}\\n```plaintext --- ## Dependency Management ### Dependency Configuration Dependencies are configured in `workspace/config/provisioning.yaml`: ```yaml\\ndependencies: # Core provisioning system core: source: \\"oci://harbor.company.com/provisioning-core:v3.5.0\\" # Extensions (providers, taskservs, clusters) extensions: source_type: \\"oci\\" oci: registry: \\"localhost:5000\\" namespace: \\"provisioning-extensions\\" tls_enabled: false auth_token_path: \\"~/.provisioning/tokens/oci\\" modules: providers: - \\"oci://localhost:5000/provisioning-extensions/aws:2.0.0\\" - \\"oci://localhost:5000/provisioning-extensions/upcloud:1.5.0\\" taskservs: - \\"oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0\\" - \\"oci://localhost:5000/provisioning-extensions/containerd:1.7.0\\" - \\"oci://localhost:5000/provisioning-extensions/etcd:3.5.0\\" clusters: - \\"oci://localhost:5000/provisioning-extensions/buildkit:0.12.0\\" # Platform services platform: source_type: \\"oci\\" oci: registry: \\"harbor.company.com\\" namespace: \\"provisioning-platform\\"\\n```plaintext ### Resolve Dependencies ```bash\\n# Resolve and install all configured dependencies\\nprovisioning dep resolve # Dry-run (show what would be installed)\\nprovisioning dep resolve --dry-run # Resolve with specific version constraints\\nprovisioning dep resolve --update # Update to latest versions\\n```plaintext ### Check for Updates ```bash\\n# Check all dependencies for updates\\nprovisioning dep check-updates # Output:\\n┬─────────────┬─────────┬────────┬──────────────────┐\\n│ name │ current │ latest │ update_available │\\n├─────────────┼─────────┼────────┼──────────────────┤\\n│ kubernetes │ 1.28.0 │ 1.29.0 │ true │\\n│ containerd │ 1.7.0 │ 1.7.0 │ false │\\n│ etcd │ 3.5.0 │ 3.5.1 │ true │\\n└─────────────┴─────────┴────────┴──────────────────┘\\n```plaintext ### Update Dependency ```bash\\n# Update specific extension to latest version\\nprovisioning dep update kubernetes # Update to specific version\\nprovisioning dep update kubernetes --version 1.29.0\\n```plaintext ### Dependency Tree ```bash\\n# Show dependency tree for extension\\nprovisioning dep tree kubernetes # Output:\\nkubernetes:1.28.0\\n├── containerd:1.7.0\\n│ └── runc:1.1.0\\n├── etcd:3.5.0\\n└── kubectl:1.28.0\\n```plaintext ### Validate Dependencies ```bash\\n# Validate dependency graph (check for cycles, conflicts)\\nprovisioning dep validate # Validate specific extension\\nprovisioning dep validate kubernetes\\n```plaintext --- ## Extension Development ### Create New Extension ```bash\\n# Generate extension from template\\nprovisioning generate extension taskserv redis # Directory structure created:\\n# extensions/taskservs/redis/\\n# ├── kcl/\\n# │ ├── kcl.mod\\n# │ ├── redis.k\\n# │ ├── version.k\\n# │ └── dependencies.k\\n# ├── scripts/\\n# │ ├── install.nu\\n# │ ├── check.nu\\n# │ └── uninstall.nu\\n# ├── templates/\\n# ├── docs/\\n# │ └── README.md\\n# ├── tests/\\n# └── manifest.yaml\\n```plaintext ### Extension Manifest Edit `manifest.yaml`: ```yaml\\nname: redis\\ntype: taskserv\\nversion: 1.0.0\\ndescription: Redis in-memory data structure store\\nauthor: Your Name\\nlicense: MIT\\nhomepage: https://redis.io\\nrepository: https://gitea.example.com/provisioning-extensions/redis dependencies: os: \\">=1.0.0\\" # Required OS taskserv tags: - database - cache - key-value platforms: - linux/amd64 - linux/arm64 min_provisioning_version: \\"3.0.0\\"\\n```plaintext ### Test Extension Locally ```bash\\n# Load extension from local path\\nprovisioning module load taskserv workspace_dev redis --source local # Test installation\\nprovisioning taskserv create redis --infra test-env --check # Run tests\\nprovisioning test extension redis\\n```plaintext ### Validate Extension ```bash\\n# Validate extension structure\\nprovisioning oci package validate ./extensions/taskservs/redis # Output:\\n✓ Extension structure valid\\nWarnings: - Missing docs/README.md (recommended)\\n```plaintext ### Package Extension ```bash\\n# Package as OCI artifact\\nprovisioning oci package ./extensions/taskservs/redis # Output: redis-1.0.0.tar.gz # Inspect package\\nprovisioning oci inspect-artifact redis-1.0.0.tar.gz\\n```plaintext ### Publish Extension ```bash\\n# Login to registry (one-time)\\nprovisioning oci login localhost:5000 # Publish extension\\nprovisioning oci push ./extensions/taskservs/redis redis 1.0.0 # Verify publication\\nprovisioning oci tags redis # Share with team\\necho \\"Published: oci://localhost:5000/provisioning-extensions/redis:1.0.0\\"\\n```plaintext --- ## Registry Setup ### Local Registry (Development) **Using Zot (lightweight)**: ```bash\\n# Start Zot registry\\nprovisioning oci-registry start # Configuration:\\n# - Endpoint: localhost:5000\\n# - Storage: ~/.provisioning/oci-registry/\\n# - No authentication\\n# - TLS disabled # Stop registry\\nprovisioning oci-registry stop # Check status\\nprovisioning oci-registry status\\n```plaintext **Manual Zot Setup**: ```bash\\n# Install Zot\\nbrew install project-zot/tap/zot # Create config\\ncat > zot-config.json <=1.7.0\\" etcd: \\"^3.5.0\\" # 3.5.x compatible\\n```plaintext ❌ **DON\'T**: Leave dependencies unversioned ```yaml\\ndependencies: containerd: \\"*\\" # Too permissive\\n```plaintext --- ### Security ✅ **DO**: - Use TLS for remote registries\\n- Rotate authentication tokens regularly\\n- Scan images for vulnerabilities (Harbor)\\n- Sign artifacts (cosign) ❌ **DON\'T**: - Use `--insecure` in production\\n- Store passwords in config files\\n- Skip certificate verification --- ## Related Documentation - [Multi-Repository Architecture](../architecture/MULTI_REPO_ARCHITECTURE.md) - Overall architecture\\n- [Extension Development Guide](extension-development.md) - Create extensions\\n- [Dependency Resolution](dependency-resolution.md) - How dependencies work\\n- OCI Client Library - Low-level API --- **Maintained By**: Documentation Team\\n**Last Updated**: 2025-10-06\\n**Next Review**: 2026-01-06","breadcrumbs":"OCI Registry Guide » Dependency Resolution Failed","id":"1684","title":"Dependency Resolution Failed"},"1685":{"body":"Date : 2025-11-23 Version : 1.0.0 For : provisioning v3.6.0+ Access powerful functionality from prov-ecosystem and provctl directly through provisioning CLI.","breadcrumbs":"Integrations Quick Start » Prov-Ecosystem & Provctl Integrations - Quick Start Guide","id":"1685","title":"Prov-Ecosystem & Provctl Integrations - Quick Start Guide"},"1686":{"body":"Four integrated feature sets: Feature Purpose Best For Runtime Abstraction Unified Docker/Podman/OrbStack/Colima/nerdctl Multi-platform deployments SSH Advanced Pooling, circuit breaker, retry strategies Large-scale distributed operations Backup System Multi-backend backups (Restic, Borg, Tar, Rsync) Data protection & disaster recovery GitOps Events Event-driven deployments from Git Continuous deployment automation Service Management Cross-platform services (systemd, launchd, runit) Infrastructure service orchestration","breadcrumbs":"Integrations Quick Start » Overview","id":"1686","title":"Overview"},"1687":{"body":"","breadcrumbs":"Integrations Quick Start » Quick Start Commands","id":"1687","title":"Quick Start Commands"},"1688":{"body":"# 1. Check what runtimes you have available\\nprovisioning runtime list # 2. Detect which runtime provisioning will use\\nprovisioning runtime detect # 3. Verify runtime works\\nprovisioning runtime info\\n```plaintext **Expected Output**: ```plaintext\\nAvailable runtimes: • docker • podman\\n```plaintext --- ## 1️⃣ Runtime Abstraction ### What It Does Automatically detects and uses Docker, Podman, OrbStack, Colima, or nerdctl - whichever is available on your system. Eliminates hardcoding \\"docker\\" commands. ### Commands ```bash\\n# Detect available runtime\\nprovisioning runtime detect\\n# Output: \\"Detected runtime: docker\\" # Execute command in runtime\\nprovisioning runtime exec \\"docker images\\"\\n# Runs: docker images # Get runtime info\\nprovisioning runtime info\\n# Shows: name, command, version # List all available runtimes\\nprovisioning runtime list\\n# Shows: docker, podman, orbstack... # Adapt docker-compose for detected runtime\\nprovisioning runtime compose ./docker-compose.yml\\n# Output: docker compose -f ./docker-compose.yml\\n```plaintext ### Examples **Use Case 1: Works on macOS with OrbStack, Linux with Docker** ```bash\\n# User on macOS with OrbStack\\n$ provisioning runtime exec \\"docker run -it ubuntu bash\\"\\n# Automatically uses orbctl (OrbStack) # User on Linux with Docker\\n$ provisioning runtime exec \\"docker run -it ubuntu bash\\"\\n# Automatically uses docker\\n```plaintext **Use Case 2: Run docker-compose with detected runtime** ```bash\\n# Detect and run compose\\n$ compose_cmd=$(provisioning runtime compose ./docker-compose.yml)\\n$ eval $compose_cmd up -d\\n# Works with docker, podman, nerdctl automatically\\n```plaintext ### Configuration No configuration needed! Runtime is auto-detected in order: 1. Docker (macOS: OrbStack first; Linux: Docker first)\\n2. Podman\\n3. OrbStack (macOS)\\n4. Colima (macOS)\\n5. nerdctl --- ## 2️⃣ SSH Advanced Operations ### What It Does Advanced SSH with connection pooling (90% faster), circuit breaker for fault isolation, and deployment strategies (rolling, blue-green, canary). ### Commands ```bash\\n# Create SSH pool connection to host\\nprovisioning ssh pool connect server.example.com root --port 22 --timeout 30 # Check pool status\\nprovisioning ssh pool status # List available deployment strategies\\nprovisioning ssh strategies\\n# Output: rolling, blue-green, canary # Configure retry strategy\\nprovisioning ssh retry-config exponential --max-retries 3 # Check circuit breaker status\\nprovisioning ssh circuit-breaker\\n# Output: state=closed, failures=0/5\\n```plaintext ### Deployment Strategies | Strategy | Use Case | Risk |\\n|----------|----------|------|\\n| **Rolling** | Gradual rollout across hosts | Low (but slower) |\\n| **Blue-Green** | Zero-downtime, instant rollback | Very low |\\n| **Canary** | Test on small % before full rollout | Very low (5% at risk) | ### Example: Multi-Host Deployment ```bash\\n# Set up SSH pool\\nprovisioning ssh pool connect srv01.example.com root\\nprovisioning ssh pool connect srv02.example.com root\\nprovisioning ssh pool connect srv03.example.com root # Execute on pool (all 3 hosts in parallel)\\nprovisioning ssh pool exec [srv01, srv02, srv03] \\"systemctl restart myapp\\" --strategy rolling # Check status\\nprovisioning ssh pool status\\n# Output: connections=3, active=0, idle=3, circuit_breaker=green\\n```plaintext ### Retry Strategies ```bash\\n# Exponential backoff: 100ms, 200ms, 400ms, 800ms...\\nprovisioning ssh retry-config exponential --max-retries 5 # Linear backoff: 100ms, 200ms, 300ms, 400ms...\\nprovisioning ssh retry-config linear --max-retries 3 # Fibonacci backoff: 100ms, 100ms, 200ms, 300ms, 500ms...\\nprovisioning ssh retry-config fibonacci --max-retries 4\\n```plaintext --- ## 3️⃣ Backup System ### What It Does Multi-backend backup management with Restic, BorgBackup, Tar, or Rsync. Supports local, S3, SFTP, REST API, and Backblaze B2 repositories. ### Commands ```bash\\n# Create backup job\\nprovisioning backup create daily-backup /data /var/lib \\\\ --backend restic \\\\ --repository s3://my-bucket/backups # Restore from snapshot\\nprovisioning backup restore snapshot-001 --restore_path /data # List available snapshots\\nprovisioning backup list # Schedule regular backups\\nprovisioning backup schedule daily-backup \\"0 2 * * *\\" \\\\ --paths [\\"/data\\" \\"/var/lib\\"] \\\\ --backend restic # Show retention policy\\nprovisioning backup retention\\n# Output: daily=7, weekly=4, monthly=12, yearly=5 # Check backup job status\\nprovisioning backup status backup-job-001\\n```plaintext ### Backend Comparison | Backend | Speed | Compression | Best For |\\n|---------|-------|-------------|----------|\\n| Restic | ⚡⚡⚡ | Excellent | Cloud backups |\\n| BorgBackup | ⚡⚡ | Excellent | Large archives |\\n| Tar | ⚡⚡⚡ | Good | Simple backups |\\n| Rsync | ⚡⚡⚡ | None | Incremental syncs | ### Example: Automated Daily Backups to S3 ```bash\\n# Create backup configuration\\nprovisioning backup create app-backup /opt/myapp /var/lib/myapp \\\\ --backend restic \\\\ --repository s3://prod-backups/myapp # Schedule daily at 2 AM\\nprovisioning backup schedule app-backup \\"0 2 * * *\\" # Set retention: keep 7 days, 4 weeks, 12 months, 5 years\\nprovisioning backup retention \\\\ --daily 7 \\\\ --weekly 4 \\\\ --monthly 12 \\\\ --yearly 5 # Verify backup was created\\nprovisioning backup list\\n```plaintext ### Dry-Run (Test First) ```bash\\n# Test backup without actually creating it\\nprovisioning backup create test-backup /data --check # Test restore without actually restoring\\nprovisioning backup restore snapshot-001 --check\\n```plaintext --- ## 4️⃣ GitOps Event-Driven Deployments ### What It Does Automatically trigger deployments from Git events (push, PR, webhook, scheduled). Supports GitHub, GitLab, Gitea. ### Commands ```bash\\n# Load GitOps rules from configuration file\\nprovisioning gitops rules ./gitops-rules.yaml # Watch for Git events (starts webhook listener)\\nprovisioning gitops watch --provider github --webhook-port 8080 # List supported events\\nprovisioning gitops events\\n# Output: push, pull-request, webhook, scheduled, health-check, manual # Manually trigger deployment\\nprovisioning gitops trigger deploy-prod --environment prod # List active deployments\\nprovisioning gitops deployments --status running # Show GitOps status\\nprovisioning gitops status\\n# Output: active_rules=5, total=42, successful=40, failed=2\\n```plaintext ### Example: GitOps Configuration **File: `gitops-rules.yaml`** ```yaml\\nrules: - name: deploy-prod provider: github repository: https://github.com/myorg/myrepo branch: main events: - push targets: - prod command: \\"provisioning deploy\\" require_approval: true - name: deploy-staging provider: github repository: https://github.com/myorg/myrepo branch: develop events: - push - pull-request targets: - staging command: \\"provisioning deploy\\" require_approval: false\\n```plaintext **Then:** ```bash\\n# Load rules\\nprovisioning gitops rules ./gitops-rules.yaml # Watch for events\\nprovisioning gitops watch --provider github # When you push to main, deployment auto-triggers!\\n# git push origin main → provisioning deploy runs automatically\\n```plaintext --- ## 5️⃣ Service Management ### What It Does Install, start, stop, and manage services across systemd (Linux), launchd (macOS), runit, and OpenRC. ### Commands ```bash\\n# Install service\\nprovisioning service install myapp /usr/local/bin/myapp \\\\ --user myapp \\\\ --working-dir /opt/myapp # Start service\\nprovisioning service start myapp # Stop service\\nprovisioning service stop myapp # Restart service\\nprovisioning service restart myapp # Check service status\\nprovisioning service status myapp\\n# Output: running=true, uptime=86400s, restarts=2 # List all services\\nprovisioning service list # Detect init system\\nprovisioning service detect-init\\n# Output: systemd (Linux), launchd (macOS), etc.\\n```plaintext ### Example: Install Custom Service ```bash\\n# On Linux (systemd)\\nprovisioning service install provisioning-worker \\\\ /usr/local/bin/provisioning-worker \\\\ --user provisioning \\\\ --working-dir /opt/provisioning # On macOS (launchd) - works the same!\\nprovisioning service install provisioning-worker \\\\ /usr/local/bin/provisioning-worker \\\\ --user provisioning \\\\ --working-dir /opt/provisioning # Service file is generated automatically for your platform\\nprovisioning service start provisioning-worker\\nprovisioning service status provisioning-worker\\n```plaintext --- ## 🎯 Common Workflows ### Workflow 1: Multi-Platform Deployment ```bash\\n# Works on macOS with OrbStack, Linux with Docker, etc.\\nprovisioning runtime detect # Detects your platform\\nprovisioning runtime exec \\"docker ps\\" # Uses your runtime\\n```plaintext ### Workflow 2: Large-Scale SSH Operations ```bash\\n# Connect to multiple servers\\nfor host in srv01 srv02 srv03; do provisioning ssh pool connect $host.example.com root\\ndone # Execute in parallel with 3x retry\\nprovisioning ssh pool exec [srv01, srv02, srv03] \\\\ \\"systemctl restart app\\" \\\\ --strategy rolling \\\\ --retry exponential\\n```plaintext ### Workflow 3: Automated Backups ```bash\\n# Create backup job\\nprovisioning backup create daily /opt/app /data \\\\ --backend restic \\\\ --repository s3://backups # Schedule for 2 AM every day\\nprovisioning backup schedule daily \\"0 2 * * *\\" # Verify it works\\nprovisioning backup list\\n```plaintext ### Workflow 4: Continuous Deployment from Git ```bash\\n# Define rules in YAML\\ncat > gitops-rules.yaml << \'EOF\'\\nrules: - name: deploy-prod provider: github repository: https://github.com/myorg/repo branch: main events: [push] targets: [prod] command: \\"provisioning deploy\\"\\nEOF # Load and activate\\nprovisioning gitops rules ./gitops-rules.yaml\\nprovisioning gitops watch --provider github # Now pushing to main auto-deploys!\\n```plaintext --- ## 🔧 Advanced Configuration ### Using with KCL Configuration All integrations support KCL schemas for advanced configuration: ```kcl\\nimport provisioning.integrations as integ # Runtime configuration\\nintegrations: integ.IntegrationConfig = { runtime = { preferred = \\"podman\\" check_order = [\\"podman\\", \\"docker\\", \\"nerdctl\\"] timeout_secs = 5 enable_cache = True } # Backup with retention policy backup = { default_backend = \\"restic\\" default_repository = { type = \\"s3\\" bucket = \\"prod-backups\\" prefix = \\"daily\\" } jobs = [] verify_after_backup = True } # GitOps rules with approval gitops = { rules = [] default_strategy = \\"blue-green\\" dry_run_by_default = False enable_audit_log = True }\\n}\\n```plaintext --- ## 💡 Tips & Tricks ### Tip 1: Dry-Run Mode All major operations support `--check` for testing: ```bash\\nprovisioning runtime exec \\"systemctl restart app\\" --check\\n# Output: Would execute: [docker exec ...] provisioning backup create test /data --check\\n# Output: Backup would be created: [test] provisioning gitops trigger deploy-test --check\\n# Output: Deployment would trigger\\n```plaintext ### Tip 2: Output Formats Some commands support JSON output: ```bash\\nprovisioning runtime list --out json\\nprovisioning backup list --out json\\nprovisioning gitops deployments --out json\\n```plaintext ### Tip 3: Integration with Scripts Chain commands in shell scripts: ```bash\\n#!/bin/bash # Detect runtime and use it\\nRUNTIME=$(provisioning runtime detect | grep -oP \'docker|podman|nerdctl\') # Execute using detected runtime\\nprovisioning runtime exec \\"docker ps\\" # Create backup before deploy\\nprovisioning backup create pre-deploy-$(date +%s) /opt/app # Deploy\\nprovisioning deploy # Verify with GitOps\\nprovisioning gitops status\\n```plaintext --- ## 🐛 Troubleshooting ### Problem: \\"No container runtime detected\\" **Solution**: Install Docker, Podman, or OrbStack: ```bash\\n# macOS\\nbrew install orbstack # Linux\\nsudo apt-get install docker.io # Then verify\\nprovisioning runtime detect\\n```plaintext ### Problem: SSH connection timeout **Solution**: Check port and timeout settings: ```bash\\n# Use different port\\nprovisioning ssh pool connect server.example.com root --port 2222 # Increase timeout\\nprovisioning ssh pool connect server.example.com root --timeout 60\\n```plaintext ### Problem: Backup fails with \\"Permission denied\\" **Solution**: Check permissions on backup path: ```bash\\n# Check if user can read target paths\\nls -l /data # Should be readable # Run with elevated privileges if needed\\nsudo provisioning backup create mybak /data --backend restic\\n```plaintext --- ## 📚 Learn More | Topic | Location |\\n|-------|----------|\\n| Architecture | `docs/architecture/ECOSYSTEM_INTEGRATION.md` |\\n| CLI Help | `provisioning help integrations` |\\n| Rust Bridge | `provisioning/platform/integrations/provisioning-bridge/` |\\n| Nushell Modules | `provisioning/core/nulib/lib_provisioning/integrations/` |\\n| KCL Schemas | `provisioning/kcl/integrations/` | --- ## 🆘 Need Help? ```bash\\n# General help\\nprovisioning help integrations # Specific command help\\nprovisioning runtime --help\\nprovisioning backup --help\\nprovisioning gitops --help # System diagnostics\\nprovisioning status\\nprovisioning health\\n```plaintext --- **Last Updated**: 2025-11-23\\n**Version**: 1.0.0","breadcrumbs":"Integrations Quick Start » 🏃 30-Second Test","id":"1688","title":"🏃 30-Second Test"},"1689":{"body":"Status : ✅ COMPLETED - All phases (1-6) implemented and tested Date : December 2025 Tests : 25/25 passing (100%)","breadcrumbs":"Secrets Service Layer Complete » Secrets Service Layer (SST) - Complete User Guide","id":"1689","title":"Secrets Service Layer (SST) - Complete User Guide"},"169":{"body":"","breadcrumbs":"First Deployment » Common Deployment Patterns","id":"169","title":"Common Deployment Patterns"},"1690":{"body":"The Secrets Service Layer (SST) is an enterprise-grade unified solution for managing all types of secrets (database credentials, SSH keys, API tokens, provider credentials) through a REST API controlled by Cedar policies with workspace isolation and real-time monitoring.","breadcrumbs":"Secrets Service Layer Complete » 📋 Executive Summary","id":"1690","title":"📋 Executive Summary"},"1691":{"body":"Feature Description Status Centralized Management Unified API for all secrets ✅ Complete Cedar Authorization Mandatory configurable policies ✅ Complete Workspace Isolation Secrets isolated by workspace and domain ✅ Complete Auto Rotation Automatic scheduling and rotation ✅ Complete Secret Sharing Cross-workspace sharing with access control ✅ Complete Real-time Monitoring Dashboard, expiration alerts ✅ Complete Complete Audit Full operation logging ✅ Complete KMS Encryption Envelope-based key encryption ✅ Complete Temporal + Permanent Support for SSH and provider credentials ✅ Complete","breadcrumbs":"Secrets Service Layer Complete » ✨ Key Features","id":"1691","title":"✨ Key Features"},"1692":{"body":"","breadcrumbs":"Secrets Service Layer Complete » 🚀 Quick Start (5 minutes)","id":"1692","title":"🚀 Quick Start (5 minutes)"},"1693":{"body":"# Register workspace\\nprovisioning workspace register librecloud /Users/Akasha/project-provisioning/workspace_librecloud # Verify\\nprovisioning workspace list\\nprovisioning workspace active\\n```plaintext ### 2. Create your first database secret ```bash\\n# Create PostgreSQL credential\\nprovisioning secrets create database postgres \\\\ --workspace librecloud \\\\ --infra wuji \\\\ --user admin \\\\ --password \\"secure_password\\" \\\\ --host db.local \\\\ --port 5432 \\\\ --database myapp\\n```plaintext ### 3. Retrieve the secret ```bash\\n# Get credential (requires Cedar authorization)\\nprovisioning secrets get librecloud/wuji/postgres/admin_password\\n```plaintext ### 4. List secrets by domain ```bash\\n# List all PostgreSQL secrets\\nprovisioning secrets list --workspace librecloud --domain postgres # List all infrastructure secrets\\nprovisioning secrets list --workspace librecloud --infra wuji\\n```plaintext --- ## 📚 Complete Guide by Phases ### Phase 1: Database and Application Secrets #### 1.1 Create Database Credentials **REST Endpoint**: ```bash\\nPOST /api/v1/secrets/database\\nContent-Type: application/json { \\"workspace_id\\": \\"librecloud\\", \\"infra_id\\": \\"wuji\\", \\"db_type\\": \\"postgresql\\", \\"host\\": \\"db.librecloud.internal\\", \\"port\\": 5432, \\"database\\": \\"production_db\\", \\"username\\": \\"admin\\", \\"password\\": \\"encrypted_password\\"\\n}\\n```plaintext **CLI Command**: ```bash\\nprovisioning secrets create database postgres \\\\ --workspace librecloud \\\\ --infra wuji \\\\ --user admin \\\\ --password \\"password\\" \\\\ --host db.librecloud.internal \\\\ --port 5432 \\\\ --database production_db\\n```plaintext **Result**: Secret stored in SurrealDB with KMS encryption ```plaintext\\n✓ Secret created: librecloud/wuji/postgres/admin_password Workspace: librecloud Infrastructure: wuji Domain: postgres Type: Database Encrypted: Yes (KMS)\\n```plaintext #### 1.2 Create Application Secrets **REST API**: ```bash\\nPOST /api/v1/secrets/application\\n{ \\"workspace_id\\": \\"librecloud\\", \\"app_name\\": \\"myapp-web\\", \\"key_type\\": \\"api_token\\", \\"value\\": \\"sk_live_abc123xyz\\"\\n}\\n```plaintext **CLI**: ```bash\\nprovisioning secrets create app myapp-web \\\\ --workspace librecloud \\\\ --domain web \\\\ --type api_token \\\\ --value \\"sk_live_abc123xyz\\"\\n```plaintext #### 1.3 List Secrets **REST API**: ```bash\\nGET /api/v1/secrets/list?workspace=librecloud&domain=postgres Response:\\n{ \\"secrets\\": [ { \\"path\\": \\"librecloud/wuji/postgres/admin_password\\", \\"workspace_id\\": \\"librecloud\\", \\"domain\\": \\"postgres\\", \\"secret_type\\": \\"Database\\", \\"created_at\\": \\"2025-12-06T10:00:00Z\\", \\"created_by\\": \\"admin\\" } ]\\n}\\n```plaintext **CLI**: ```bash\\n# All workspace secrets\\nprovisioning secrets list --workspace librecloud # Filter by domain\\nprovisioning secrets list --workspace librecloud --domain postgres # Filter by infrastructure\\nprovisioning secrets list --workspace librecloud --infra wuji\\n```plaintext #### 1.4 Retrieve a Secret **REST API**: ```bash\\nGET /api/v1/secrets/librecloud/wuji/postgres/admin_password Requires:\\n- Header: Authorization: Bearer \\n- Cedar verification: [user has read permission]\\n- If MFA required: mfa_verified=true in JWT\\n```plaintext **CLI**: ```bash\\n# Get full secret\\nprovisioning secrets get librecloud/wuji/postgres/admin_password # Output:\\n# Host: db.librecloud.internal\\n# Port: 5432\\n# User: admin\\n# Database: production_db\\n# Password: [encrypted in transit]\\n```plaintext --- ### Phase 2: SSH Keys and Provider Credentials #### 2.1 Temporal SSH Keys (Auto-expiring) **Use Case**: Temporary server access (max 24 hours) ```bash\\n# Generate temporary SSH key (TTL 2 hours)\\nprovisioning secrets create ssh \\\\ --workspace librecloud \\\\ --infra wuji \\\\ --server web01 \\\\ --ttl 2h # Result:\\n# ✓ SSH key generated\\n# Server: web01\\n# TTL: 2 hours\\n# Expires at: 2025-12-06T12:00:00Z\\n# Private Key: [encrypted]\\n```plaintext **Technical Details**: - Generated in real-time by Orchestrator\\n- Stored in memory (TTL-based)\\n- Automatic revocation on expiry\\n- Complete audit trail in vault_audit #### 2.2 Permanent SSH Keys (Stored) **Use Case**: Long-duration infrastructure keys ```bash\\n# Create permanent SSH key (stored in DB)\\nprovisioning secrets create ssh \\\\ --workspace librecloud \\\\ --infra wuji \\\\ --server web01 \\\\ --permanent # Result:\\n# ✓ Permanent SSH key created\\n# Storage: SurrealDB (encrypted)\\n# Rotation: Manual (or automatic if configured)\\n# Access: Cedar controlled\\n```plaintext #### 2.3 Provider Credentials **UpCloud API (Temporal)**: ```bash\\nprovisioning secrets create provider upcloud \\\\ --workspace librecloud \\\\ --roles \\"server,network,storage\\" \\\\ --ttl 4h # Result:\\n# ✓ UpCloud credential generated\\n# Token: tmp_upcloud_abc123\\n# Roles: server, network, storage\\n# TTL: 4 hours\\n```plaintext **UpCloud API (Permanent)**: ```bash\\nprovisioning secrets create provider upcloud \\\\ --workspace librecloud \\\\ --roles \\"server,network\\" \\\\ --permanent # Result:\\n# ✓ Permanent UpCloud credential created\\n# Token: upcloud_live_xyz789\\n# Storage: SurrealDB\\n# Rotation: Manual\\n```plaintext --- ### Phase 3: Auto Rotation #### 3.1 Plan Automatic Rotation **Predefined Rotation Policies**: | Type | Prod | Dev |\\n|------|------|-----|\\n| **Database** | Every 30d | Every 90d |\\n| **Application** | Every 60d | Every 14d |\\n| **SSH** | Every 365d | Every 90d |\\n| **Provider** | Every 180d | Every 30d | **Force Immediate Rotation**: ```bash\\n# Force rotation now\\nprovisioning secrets rotate librecloud/wuji/postgres/admin_password # Result:\\n# ✓ Rotation initiated\\n# Status: In Progress\\n# New password: [generated]\\n# Old password: [archived]\\n# Next rotation: 2025-01-05\\n```plaintext **Check Rotation Status**: ```bash\\nGET /api/v1/secrets/{path}/rotation-status Response:\\n{ \\"path\\": \\"librecloud/wuji/postgres/admin_password\\", \\"status\\": \\"pending\\", \\"next_rotation\\": \\"2025-01-05T10:00:00Z\\", \\"last_rotation\\": \\"2025-12-05T10:00:00Z\\", \\"days_remaining\\": 30, \\"failure_count\\": 0\\n}\\n```plaintext #### 3.2 Rotation Job Scheduler (Background) System automatically runs rotations every hour: ```plaintext\\n┌─────────────────────────────────┐\\n│ Rotation Job Scheduler │\\n│ - Interval: 1 hour │\\n│ - Max concurrency: 5 rotations │\\n│ - Auto retry │\\n└─────────────────────────────────┘ ↓ Get due secrets ↓ Generate new credentials ↓ Validate functionality ↓ Update SurrealDB ↓ Log to audit trail\\n```plaintext **Check Scheduler Status**: ```bash\\nprovisioning secrets scheduler status # Result:\\n# Status: Running\\n# Last check: 2025-12-06T11:00:00Z\\n# Completed rotations: 24\\n# Failed rotations: 0\\n```plaintext --- ### Phase 3.2: Share Secrets Across Workspaces #### Create a Grant (Access Authorization) **Scenario**: Share DB credential between `librecloud` and `staging` ```bash\\n# REST API\\nPOST /api/v1/secrets/{path}/grant { \\"source_workspace\\": \\"librecloud\\", \\"target_workspace\\": \\"staging\\", \\"permission\\": \\"read\\", # read, write, rotate \\"require_approval\\": false\\n} # Response:\\n{ \\"grant_id\\": \\"grant-12345\\", \\"secret_path\\": \\"librecloud/wuji/postgres/admin_password\\", \\"source_workspace\\": \\"librecloud\\", \\"target_workspace\\": \\"staging\\", \\"permission\\": \\"read\\", \\"status\\": \\"active\\", \\"granted_at\\": \\"2025-12-06T10:00:00Z\\", \\"access_count\\": 0\\n}\\n```plaintext **CLI**: ```bash\\nprovisioning secrets grant \\\\ --secret librecloud/wuji/postgres/admin_password \\\\ --target-workspace staging \\\\ --permission read # ✓ Grant created: grant-12345\\n# Source workspace: librecloud\\n# Target workspace: staging\\n# Permission: Read\\n# Approval required: No\\n```plaintext #### Revoke a Grant ```bash\\n# Revoke access immediately\\nPOST /api/v1/secrets/grant/{grant_id}/revoke\\n{ \\"reason\\": \\"User left the team\\"\\n} # CLI\\nprovisioning secrets revoke-grant grant-12345 \\\\ --reason \\"User left the team\\" # ✓ Grant revoked\\n# Status: Revoked\\n# Access records: 42\\n```plaintext #### List Grants ```bash\\n# All workspace grants\\nGET /api/v1/secrets/grants?workspace=librecloud # Response:\\n{ \\"grants\\": [ { \\"grant_id\\": \\"grant-12345\\", \\"secret_path\\": \\"librecloud/wuji/postgres/admin_password\\", \\"target_workspace\\": \\"staging\\", \\"permission\\": \\"read\\", \\"status\\": \\"active\\", \\"access_count\\": 42, \\"last_accessed\\": \\"2025-12-06T10:30:00Z\\" } ]\\n}\\n```plaintext --- ### Phase 3.4: Monitoring and Alerts #### Dashboard Metrics ```bash\\nGET /api/v1/secrets/monitoring/dashboard Response:\\n{ \\"total_secrets\\": 45, \\"temporal_secrets\\": 12, \\"permanent_secrets\\": 33, \\"expiring_secrets\\": [ { \\"path\\": \\"librecloud/wuji/postgres/admin_password\\", \\"domain\\": \\"postgres\\", \\"days_remaining\\": 5, \\"severity\\": \\"critical\\" } ], \\"failed_access_attempts\\": [ { \\"user\\": \\"alice\\", \\"secret_path\\": \\"librecloud/wuji/postgres/admin_password\\", \\"reason\\": \\"insufficient_permissions\\", \\"timestamp\\": \\"2025-12-06T10:00:00Z\\" } ], \\"rotation_metrics\\": { \\"total\\": 45, \\"completed\\": 40, \\"pending\\": 3, \\"failed\\": 2 }\\n}\\n```plaintext **CLI**: ```bash\\nprovisioning secrets monitoring dashboard # ✓ Secrets Dashboard - Librecloud\\n#\\n# Total secrets: 45\\n# Temporal secrets: 12\\n# Permanent secrets: 33\\n#\\n# ⚠️ CRITICAL (next 3 days): 2\\n# - librecloud/wuji/postgres/admin_password (5 days)\\n# - librecloud/wuji/redis/password (1 day)\\n#\\n# ⚡ WARNING (next 7 days): 3\\n# - librecloud/app/api_token (7 days)\\n#\\n# 📊 Rotations completed: 40/45 (89%)\\n```plaintext #### Expiring Secrets Alerts ```bash\\nGET /api/v1/secrets/monitoring/expiring?days=7 Response:\\n{ \\"expiring_secrets\\": [ { \\"path\\": \\"librecloud/wuji/postgres/admin_password\\", \\"domain\\": \\"postgres\\", \\"expires_in_days\\": 5, \\"type\\": \\"database\\", \\"last_rotation\\": \\"2025-11-05T10:00:00Z\\" } ]\\n}\\n```plaintext --- ## 🔐 Cedar Authorization All operations are protected by **Cedar policies**: ### Example Policy: Production Secret Access ```cedar\\n// Requires MFA for production secrets\\n@id(\\"prod-secret-access-mfa\\")\\npermit ( principal, action == Provisioning::Action::\\"access\\", resource is Provisioning::Secret in Provisioning::Environment::\\"production\\"\\n) when { context.mfa_verified == true && resource.is_expired == false\\n}; // Only admins can create permanent secrets\\n@id(\\"permanent-secret-admin-only\\")\\npermit ( principal in Provisioning::Role::\\"security_admin\\", action == Provisioning::Action::\\"create\\", resource is Provisioning::Secret\\n) when { resource.lifecycle == \\"permanent\\"\\n};\\n```plaintext ### Verify Authorization ```bash\\n# Test Cedar decision\\nprovisioning policies check alice can access secret:librecloud/postgres/password # Result:\\n# User: alice\\n# Resource: secret:librecloud/postgres/password\\n# Decision: ✅ ALLOWED\\n# - Role: database_admin\\n# - MFA verified: Yes\\n# - Workspace: librecloud\\n```plaintext --- ## 🏗️ Data Structure ### Secret in Database ```sql\\n-- Table vault_secrets (SurrealDB)\\n{ id: \\"secret:uuid123\\", path: \\"librecloud/wuji/postgres/admin_password\\", workspace_id: \\"librecloud\\", infra_id: \\"wuji\\", domain: \\"postgres\\", secret_type: \\"Database\\", encrypted_value: \\"U2FsdGVkX1...\\", -- AES-256-GCM encrypted version: 1, created_at: \\"2025-12-05T10:00:00Z\\", created_by: \\"admin\\", updated_at: \\"2025-12-05T10:00:00Z\\", updated_by: \\"admin\\", tags: [\\"production\\", \\"critical\\"], auto_rotate: true, rotation_interval_days: 30, ttl_seconds: null, -- null = no auto expiry deleted: false, metadata: { db_host: \\"db.librecloud.internal\\", db_port: 5432, db_name: \\"production_db\\", username: \\"admin\\" }\\n}\\n```plaintext ### Secret Hierarchy ```plaintext\\nlibrecloud (Workspace) ├── wuji (Infrastructure) │ ├── postgres (Domain) │ │ ├── admin_password │ │ ├── readonly_user │ │ └── replication_user │ ├── redis (Domain) │ │ └── master_password │ └── ssh (Domain) │ ├── web01_key │ └── db01_key └── web (Infrastructure) ├── api (Domain) │ ├── stripe_token │ ├── github_token │ └── sendgrid_key └── auth (Domain) ├── jwt_secret └── oauth_client_secret\\n```plaintext --- ## 🔄 Complete Workflows ### Workflow 1: Create and Rotate Database Credential ```plaintext\\n1. Admin creates credential POST /api/v1/secrets/database 2. System encrypts with KMS ├─ Generates data key ├─ Encrypts secret with data key └─ Encrypts data key with KMS master key 3. Stores in SurrealDB ├─ vault_secrets (encrypted value) ├─ vault_versions (history) └─ vault_audit (audit record) 4. System schedules auto rotation ├─ Calculates next date (30 days) └─ Creates rotation_scheduler entry 5. Every hour, background job checks ├─ Any secrets due for rotation? ├─ Yes → Generate new password ├─ Validate functionality (connect to DB) ├─ Update SurrealDB └─ Log to audit 6. Monitoring alerts ├─ If 7 days remaining → WARNING alert ├─ If 3 days remaining → CRITICAL alert └─ If expired → EXPIRED alert\\n```plaintext ### Workflow 2: Share Secret Between Workspaces ```plaintext\\n1. Admin of librecloud creates grant POST /api/v1/secrets/{path}/grant 2. Cedar verifies authorization ├─ Is user admin of source workspace? └─ Is target workspace valid? 3. Grant created and recorded ├─ Unique ID: grant-xxxxx ├─ Status: active └─ Audit: who, when, why 4. Staging workspace user accesses secret GET /api/v1/secrets/{path} 5. System verifies access ├─ Cedar: Is grant active? ├─ Cedar: Sufficient permission? ├─ Cedar: MFA if required? └─ Yes → Return decrypted secret 6. Audit records access ├─ User who accessed ├─ Source IP ├─ Exact timestamp ├─ Success/failure └─ Increment access count in grant\\n```plaintext ### Workflow 3: Access Temporal SSH Secret ```plaintext\\n1. User requests temporary SSH key POST /api/v1/secrets/ssh {ttl: \\"2h\\"} 2. Cedar authorizes (requires MFA) ├─ User has role? ├─ MFA verified? └─ TTL within limit (max 24h)? 3. Orchestrator generates key ├─ Generates SSH key pair (RSA 4096) ├─ Stores in memory (TTL-based) ├─ Logs to audit └─ Returns private key 4. User downloads key └─ Valid for 2 hours 5. Automatic expiration ├─ 2-hour timer starts ├─ TTL expires → Auto revokes ├─ Later attempts → Access denied └─ Audit: automatic revocation\\n```plaintext --- ## 📝 Practical Examples ### Example 1: Manage PostgreSQL Secrets ```bash\\n# 1. Create credential\\nprovisioning secrets create database postgres \\\\ --workspace librecloud \\\\ --infra wuji \\\\ --user admin \\\\ --password \\"P@ssw0rd123!\\" \\\\ --host db.librecloud.internal \\\\ --port 5432 \\\\ --database myapp_prod # 2. List PostgreSQL secrets\\nprovisioning secrets list --workspace librecloud --domain postgres # 3. Get for connection\\nprovisioning secrets get librecloud/wuji/postgres/admin_password # 4. Share with staging team\\nprovisioning secrets grant \\\\ --secret librecloud/wuji/postgres/admin_password \\\\ --target-workspace staging \\\\ --permission read # 5. Force rotation\\nprovisioning secrets rotate librecloud/wuji/postgres/admin_password # 6. Check status\\nprovisioning secrets monitoring dashboard | grep postgres\\n```plaintext ### Example 2: Temporary SSH Access ```bash\\n# 1. Generate temporary SSH key (4 hours)\\nprovisioning secrets create ssh \\\\ --workspace librecloud \\\\ --infra wuji \\\\ --server web01 \\\\ --ttl 4h # 2. Download private key\\nprovisioning secrets get librecloud/wuji/ssh/web01_key > ~/.ssh/web01_temp # 3. Connect to server\\nchmod 600 ~/.ssh/web01_temp\\nssh -i ~/.ssh/web01_temp ubuntu@web01.librecloud.internal # 4. After 4 hours\\n# → Key revoked automatically\\n# → New SSH attempts fail\\n# → Access logged in audit\\n```plaintext ### Example 3: CI/CD Integration ```yaml\\n# GitLab CI / GitHub Actions\\njobs: deploy: script: # 1. Get DB credential - export DB_PASSWORD=$(provisioning secrets get librecloud/prod/postgres/admin_password) # 2. Get API token - export API_TOKEN=$(provisioning secrets get librecloud/app/api_token) # 3. Deploy application - docker run -e DB_PASSWORD=$DB_PASSWORD -e API_TOKEN=$API_TOKEN myapp:latest # 4. System logs access in audit # → User: ci-deploy # → Workspace: librecloud # → Secrets accessed: 2 # → Status: success\\n```plaintext --- ## 🛡️ Security ### Encryption - **At Rest**: AES-256-GCM with KMS key rotation\\n- **In Transit**: TLS 1.3\\n- **In Memory**: Automatic cleanup of sensitive variables ### Access Control - **Cedar**: All operations evaluated against policies\\n- **MFA**: Required for production secrets\\n- **Workspace Isolation**: Data separation at DB level ### Audit ```json\\n{ \\"timestamp\\": \\"2025-12-06T10:30:45Z\\", \\"user_id\\": \\"alice\\", \\"workspace\\": \\"librecloud\\", \\"action\\": \\"secrets:get\\", \\"resource\\": \\"librecloud/wuji/postgres/admin_password\\", \\"result\\": \\"success\\", \\"ip_address\\": \\"192.168.1.100\\", \\"mfa_verified\\": true, \\"cedar_policy\\": \\"prod-secret-access-mfa\\"\\n}\\n```plaintext --- ## 📊 Test Results ### All 25 Integration Tests Passing ```plaintext\\n✅ Phase 3.1: Rotation Scheduler (9 tests) - Schedule creation - Status transitions - Failure tracking ✅ Phase 3.2: Secret Sharing (8 tests) - Grant creation with permissions - Permission hierarchy - Access logging ✅ Phase 3.4: Monitoring (4 tests) - Dashboard metrics - Expiring alerts - Failed access recording ✅ Phase 5: Rotation Job Scheduler (4 tests) - Background job lifecycle - Configuration management ✅ Integration Tests (3 tests) - Multi-service workflows - End-to-end scenarios\\n```plaintext **Execution**: ```bash\\ncargo test --test secrets_phases_integration_test test result: ok. 25 passed; 0 failed\\n```plaintext --- ## 🆘 Troubleshooting ### Problem: \\"Authorization denied by Cedar policy\\" **Cause**: User lacks permissions in policy\\n**Solution**: ```bash\\n# Check user and permission\\nprovisioning policies check $USER can access secret:librecloud/postgres/admin_password # Check roles\\nprovisioning auth whoami # Request access from admin\\nprovisioning secrets grant \\\\ --secret librecloud/wuji/postgres/admin_password \\\\ --target-workspace $WORKSPACE \\\\ --permission read\\n```plaintext ### Problem: \\"Secret not found\\" **Cause**: Typo in path or workspace doesn\'t exist\\n**Solution**: ```bash\\n# List available secrets\\nprovisioning secrets list --workspace librecloud # Check active workspace\\nprovisioning workspace active # Switch workspace if needed\\nprovisioning workspace switch librecloud\\n```plaintext ### Problem: \\"MFA required\\" **Cause**: Operation requires MFA but not verified\\n**Solution**: ```bash\\n# Check MFA status\\nprovisioning auth status # Enroll if not configured\\nprovisioning mfa totp enroll # Use MFA token on next access\\nprovisioning secrets get librecloud/wuji/postgres/admin_password --mfa-code 123456\\n```plaintext --- ## 📚 Complete Documentation - **REST API**: `/docs/api/secrets-api.md`\\n- **CLI Reference**: `provisioning secrets --help`\\n- **Cedar Policies**: `provisioning/config/cedar-policies/secrets.cedar`\\n- **Architecture**: `/docs/architecture/SECRETS_SERVICE_LAYER.md`\\n- **Security**: `/docs/user/SECRETS_SECURITY_GUIDE.md` --- ## 🎯 Next Steps (Future) 1. **Phase 7**: Web UI Dashboard for visual management\\n2. **Phase 8**: HashiCorp Vault integration\\n3. **Phase 9**: Multi-datacenter secret replication --- **Status**: ✅ Secrets Service Layer - COMPLETED AND TESTED","breadcrumbs":"Secrets Service Layer Complete » 1. Register the workspace librecloud","id":"1693","title":"1. Register the workspace librecloud"},"1694":{"body":"Comprehensive OCI (Open Container Initiative) registry deployment and management for the provisioning system. Source : provisioning/platform/oci-registry/","breadcrumbs":"OCI Registry Platform » OCI Registry Service","id":"1694","title":"OCI Registry Service"},"1695":{"body":"Zot (Recommended for Development): Lightweight, fast, OCI-native with UI Harbor (Recommended for Production): Full-featured enterprise registry Distribution (OCI Reference): Official OCI reference implementation","breadcrumbs":"OCI Registry Platform » Supported Registries","id":"1695","title":"Supported Registries"},"1696":{"body":"Multi-Registry Support : Zot, Harbor, Distribution Namespace Organization : Logical separation of artifacts Access Control : RBAC, policies, authentication Monitoring : Prometheus metrics, health checks Garbage Collection : Automatic cleanup of unused artifacts High Availability : Optional HA configurations TLS/SSL : Secure communication UI Interface : Web-based management (Zot, Harbor)","breadcrumbs":"OCI Registry Platform » Features","id":"1696","title":"Features"},"1697":{"body":"","breadcrumbs":"OCI Registry Platform » Quick Start","id":"1697","title":"Quick Start"},"1698":{"body":"cd provisioning/platform/oci-registry/zot\\ndocker-compose up -d # Initialize with namespaces and policies\\nnu ../scripts/init-registry.nu --registry-type zot # Access UI\\nopen http://localhost:5000","breadcrumbs":"OCI Registry Platform » Start Zot Registry (Default)","id":"1698","title":"Start Zot Registry (Default)"},"1699":{"body":"cd provisioning/platform/oci-registry/harbor\\ndocker-compose up -d\\nsleep 120 # Wait for services # Initialize\\nnu ../scripts/init-registry.nu --registry-type harbor --admin-password Harbor12345 # Access UI\\nopen http://localhost\\n# Login: admin / Harbor12345","breadcrumbs":"OCI Registry Platform » Start Harbor Registry","id":"1699","title":"Start Harbor Registry"},"17":{"body":"Component Minimum Recommended CPU 2 cores 4+ cores RAM 4 GB 8+ GB Storage 2 GB free 10+ GB free Network Internet connection Broadband connection","breadcrumbs":"Installation Guide » Hardware Requirements","id":"17","title":"Hardware Requirements"},"170":{"body":"Create multiple servers at once: servers = [ {hostname = \\"web-01\\", cores = 2, memory = 4096}, {hostname = \\"web-02\\", cores = 2, memory = 4096}, {hostname = \\"db-01\\", cores = 4, memory = 8192}\\n] provisioning server create --infra my-infra --servers web-01,web-02,db-01","breadcrumbs":"First Deployment » Pattern 1: Multiple Servers","id":"170","title":"Pattern 1: Multiple Servers"},"1700":{"body":"Namespace Description Public Retention provisioning-extensions Extension packages No 10 tags, 90 days provisioning-kcl KCL schemas No 20 tags, 180 days provisioning-platform Platform images No 5 tags, 30 days provisioning-test Test artifacts Yes 3 tags, 7 days","breadcrumbs":"OCI Registry Platform » Default Namespaces","id":"1700","title":"Default Namespaces"},"1701":{"body":"","breadcrumbs":"OCI Registry Platform » Management","id":"1701","title":"Management"},"1702":{"body":"# Start registry\\nnu -c \\"use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry start --type zot\\" # Check status\\nnu -c \\"use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry status --type zot\\" # View logs\\nnu -c \\"use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry logs --type zot --follow\\" # Health check\\nnu -c \\"use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry health --type zot\\" # List namespaces\\nnu -c \\"use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry namespaces\\"","breadcrumbs":"OCI Registry Platform » Nushell Commands","id":"1702","title":"Nushell Commands"},"1703":{"body":"# Start\\ndocker-compose up -d # Stop\\ndocker-compose down # View logs\\ndocker-compose logs -f # Remove (including volumes)\\ndocker-compose down -v","breadcrumbs":"OCI Registry Platform » Docker Compose","id":"1703","title":"Docker Compose"},"1704":{"body":"Feature Zot Harbor Distribution Setup Simple Complex Simple UI Built-in Full-featured None Search Yes Yes No Scanning No Trivy No Replication No Yes No RBAC Basic Advanced Basic Best For Dev/CI Production Compliance","breadcrumbs":"OCI Registry Platform » Registry Comparison","id":"1704","title":"Registry Comparison"},"1705":{"body":"","breadcrumbs":"OCI Registry Platform » Security","id":"1705","title":"Security"},"1706":{"body":"Zot/Distribution (htpasswd) : htpasswd -Bc htpasswd provisioning\\ndocker login localhost:5000 Harbor (Database) : docker login localhost\\n# Username: admin / Password: Harbor12345","breadcrumbs":"OCI Registry Platform » Authentication","id":"1706","title":"Authentication"},"1707":{"body":"","breadcrumbs":"OCI Registry Platform » Monitoring","id":"1707","title":"Monitoring"},"1708":{"body":"# API check\\ncurl http://localhost:5000/v2/ # Catalog check\\ncurl http://localhost:5000/v2/_catalog","breadcrumbs":"OCI Registry Platform » Health Checks","id":"1708","title":"Health Checks"},"1709":{"body":"Zot : curl http://localhost:5000/metrics Harbor : curl http://localhost:9090/metrics","breadcrumbs":"OCI Registry Platform » Metrics","id":"1709","title":"Metrics"},"171":{"body":"Install multiple services on one server: provisioning taskserv create kubernetes,cilium,postgres --infra my-infra --servers web-01","breadcrumbs":"First Deployment » Pattern 2: Server with Multiple Task Services","id":"171","title":"Pattern 2: Server with Multiple Task Services"},"1710":{"body":"Architecture : OCI Integration User Guide : OCI Registry Guide","breadcrumbs":"OCI Registry Platform » Related Documentation","id":"1710","title":"Related Documentation"},"1711":{"body":"Version : 1.0.0 Date : 2025-10-06 Status : Production Ready","breadcrumbs":"Test Environment Guide » Test Environment Guide","id":"1711","title":"Test Environment Guide"},"1712":{"body":"The Test Environment Service provides automated containerized testing for taskservs, servers, and multi-node clusters. Built into the orchestrator, it eliminates manual Docker management and provides realistic test scenarios.","breadcrumbs":"Test Environment Guide » Overview","id":"1712","title":"Overview"},"1713":{"body":"┌─────────────────────────────────────────────────┐\\n│ Orchestrator (port 8080) │\\n│ ┌──────────────────────────────────────────┐ │\\n│ │ Test Orchestrator │ │\\n│ │ • Container Manager (Docker API) │ │\\n│ │ • Network Isolation │ │\\n│ │ • Multi-node Topologies │ │\\n│ │ • Test Execution │ │\\n│ └──────────────────────────────────────────┘ │\\n└─────────────────────────────────────────────────┘ ↓ ┌────────────────────────┐ │ Docker Containers │ │ • Isolated Networks │ │ • Resource Limits │ │ • Volume Mounts │ └────────────────────────┘\\n```plaintext ## Test Environment Types ### 1. Single Taskserv Test Test individual taskserv in isolated container. ```bash\\n# Basic test\\nprovisioning test env single kubernetes # With resource limits\\nprovisioning test env single redis --cpu 2000 --memory 4096 # Auto-start and cleanup\\nprovisioning test quick postgres\\n```plaintext ### 2. Server Simulation Simulate complete server with multiple taskservs. ```bash\\n# Server with taskservs\\nprovisioning test env server web-01 [containerd kubernetes cilium] # With infrastructure context\\nprovisioning test env server db-01 [postgres redis] --infra prod-stack\\n```plaintext ### 3. Cluster Topology Multi-node cluster simulation from templates. ```bash\\n# 3-node Kubernetes cluster\\nprovisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start # etcd cluster\\nprovisioning test topology load etcd_cluster | test env cluster etcd\\n```plaintext ## Quick Start ### Prerequisites 1. **Docker running:** ```bash docker ps # Should work without errors Orchestrator running: cd provisioning/platform/orchestrator\\n./scripts/start-orchestrator.nu --background","breadcrumbs":"Test Environment Guide » Architecture","id":"1713","title":"Architecture"},"1714":{"body":"# 1. Quick test (fastest)\\nprovisioning test quick kubernetes # 2. Or step-by-step\\n# Create environment\\nprovisioning test env single kubernetes --auto-start # List environments\\nprovisioning test env list # Check status\\nprovisioning test env status  # View logs\\nprovisioning test env logs  # Cleanup\\nprovisioning test env cleanup \\n```plaintext ## Topology Templates ### Available Templates ```bash\\n# List templates\\nprovisioning test topology list\\n```plaintext | Template | Description | Nodes |\\n|----------|-------------|-------|\\n| `kubernetes_3node` | K8s HA cluster | 1 CP + 2 workers |\\n| `kubernetes_single` | All-in-one K8s | 1 node |\\n| `etcd_cluster` | etcd cluster | 3 members |\\n| `containerd_test` | Standalone containerd | 1 node |\\n| `postgres_redis` | Database stack | 2 nodes | ### Using Templates ```bash\\n# Load and use template\\nprovisioning test topology load kubernetes_3node | test env cluster kubernetes # View template\\nprovisioning test topology load etcd_cluster\\n```plaintext ### Custom Topology Create `my-topology.toml`: ```toml\\n[my_cluster]\\nname = \\"My Custom Cluster\\"\\ncluster_type = \\"custom\\" [[my_cluster.nodes]]\\nname = \\"node-01\\"\\nrole = \\"primary\\"\\ntaskservs = [\\"postgres\\", \\"redis\\"]\\n[my_cluster.nodes.resources]\\ncpu_millicores = 2000\\nmemory_mb = 4096 [[my_cluster.nodes]]\\nname = \\"node-02\\"\\nrole = \\"replica\\"\\ntaskservs = [\\"postgres\\"]\\n[my_cluster.nodes.resources]\\ncpu_millicores = 1000\\nmemory_mb = 2048 [my_cluster.network]\\nsubnet = \\"172.30.0.0/16\\"\\n```plaintext ## Commands Reference ### Environment Management ```bash\\n# Create from config\\nprovisioning test env create  # Single taskserv\\nprovisioning test env single  [--cpu N] [--memory MB] # Server simulation\\nprovisioning test env server   [--infra NAME] # Cluster topology\\nprovisioning test env cluster   # List environments\\nprovisioning test env list # Get details\\nprovisioning test env get  # Show status\\nprovisioning test env status \\n```plaintext ### Test Execution ```bash\\n# Run tests\\nprovisioning test env run  [--tests [test1, test2]] # View logs\\nprovisioning test env logs  # Cleanup\\nprovisioning test env cleanup \\n```plaintext ### Quick Test ```bash\\n# One-command test (create, run, cleanup)\\nprovisioning test quick  [--infra NAME]\\n```plaintext ## REST API ### Create Environment ```bash\\ncurl -X POST http://localhost:9090/test/environments/create \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{ \\"config\\": { \\"type\\": \\"single_taskserv\\", \\"taskserv\\": \\"kubernetes\\", \\"base_image\\": \\"ubuntu:22.04\\", \\"environment\\": {}, \\"resources\\": { \\"cpu_millicores\\": 2000, \\"memory_mb\\": 4096 } }, \\"infra\\": \\"my-project\\", \\"auto_start\\": true, \\"auto_cleanup\\": false }\'\\n```plaintext ### List Environments ```bash\\ncurl http://localhost:9090/test/environments\\n```plaintext ### Run Tests ```bash\\ncurl -X POST http://localhost:9090/test/environments/{id}/run \\\\ -H \\"Content-Type: application/json\\" \\\\ -d \'{ \\"tests\\": [], \\"timeout_seconds\\": 300 }\'\\n```plaintext ### Cleanup ```bash\\ncurl -X DELETE http://localhost:9090/test/environments/{id}\\n```plaintext ## Use Cases ### 1. Taskserv Development Test taskserv before deployment: ```bash\\n# Test new taskserv version\\nprovisioning test env single my-taskserv --auto-start # Check logs\\nprovisioning test env logs \\n```plaintext ### 2. Multi-Taskserv Integration Test taskserv combinations: ```bash\\n# Test kubernetes + cilium + containerd\\nprovisioning test env server k8s-test [kubernetes cilium containerd] --auto-start\\n```plaintext ### 3. Cluster Validation Test cluster configurations: ```bash\\n# Test 3-node etcd cluster\\nprovisioning test topology load etcd_cluster | test env cluster etcd --auto-start\\n```plaintext ### 4. CI/CD Integration ```yaml\\n# .gitlab-ci.yml\\ntest-taskserv: stage: test script: - provisioning test quick kubernetes - provisioning test quick redis - provisioning test quick postgres\\n```plaintext ## Advanced Features ### Resource Limits ```bash\\n# Custom CPU and memory\\nprovisioning test env single postgres \\\\ --cpu 4000 \\\\ --memory 8192\\n```plaintext ### Network Isolation Each environment gets isolated network: - Subnet: 172.20.0.0/16 (default)\\n- DNS enabled\\n- Container-to-container communication ### Auto-Cleanup ```bash\\n# Auto-cleanup after tests\\nprovisioning test env single redis --auto-start --auto-cleanup\\n```plaintext ### Multiple Environments Run tests in parallel: ```bash\\n# Create multiple environments\\nprovisioning test env single kubernetes --auto-start &\\nprovisioning test env single postgres --auto-start &\\nprovisioning test env single redis --auto-start & wait # List all\\nprovisioning test env list\\n```plaintext ## Troubleshooting ### Docker not running ```plaintext\\nError: Failed to connect to Docker\\n```plaintext **Solution:** ```bash\\n# Check Docker\\ndocker ps # Start Docker daemon\\nsudo systemctl start docker # Linux\\nopen -a Docker # macOS\\n```plaintext ### Orchestrator not running ```plaintext\\nError: Connection refused (port 8080)\\n```plaintext **Solution:** ```bash\\ncd provisioning/platform/orchestrator\\n./scripts/start-orchestrator.nu --background\\n```plaintext ### Environment creation fails Check logs: ```bash\\nprovisioning test env logs \\n```plaintext Check Docker: ```bash\\ndocker ps -a\\ndocker logs \\n```plaintext ### Out of resources ```plaintext\\nError: Cannot allocate memory\\n```plaintext **Solution:** ```bash\\n# Cleanup old environments\\nprovisioning test env list | each {|env| provisioning test env cleanup $env.id } # Or cleanup Docker\\ndocker system prune -af\\n```plaintext ## Best Practices ### 1. Use Templates Reuse topology templates instead of recreating: ```bash\\nprovisioning test topology load kubernetes_3node | test env cluster kubernetes\\n```plaintext ### 2. Auto-Cleanup Always use auto-cleanup in CI/CD: ```bash\\nprovisioning test quick  # Includes auto-cleanup\\n```plaintext ### 3. Resource Planning Adjust resources based on needs: - Development: 1-2 cores, 2GB RAM\\n- Integration: 2-4 cores, 4-8GB RAM\\n- Production-like: 4+ cores, 8+ GB RAM ### 4. Parallel Testing Run independent tests in parallel: ```bash\\nfor taskserv in [kubernetes postgres redis] { provisioning test quick $taskserv &\\n}\\nwait\\n```plaintext ## Configuration ### Default Settings - Base image: `ubuntu:22.04`\\n- CPU: 1000 millicores (1 core)\\n- Memory: 2048 MB (2GB)\\n- Network: 172.20.0.0/16 ### Custom Config ```bash\\n# Override defaults\\nprovisioning test env single postgres \\\\ --base-image debian:12 \\\\ --cpu 2000 \\\\ --memory 4096\\n```plaintext --- ## Related Documentation - [Test Environment API](../api/test-environment-api.md)\\n- [Topology Templates](../architecture/test-topologies.md)\\n- [Orchestrator Guide](orchestrator-guide.md)\\n- [Taskserv Development](taskserv-development.md) --- ## Version History | Version | Date | Changes |\\n|---------|------|---------|\\n| 1.0.0 | 2025-10-06 | Initial test environment service | --- **Maintained By**: Infrastructure Team","breadcrumbs":"Test Environment Guide » Basic Workflow","id":"1714","title":"Basic Workflow"},"1715":{"body":"","breadcrumbs":"Test Environment Usage » Test Environment Usage","id":"1715","title":"Test Environment Usage"},"1716":{"body":"","breadcrumbs":"Test Environment System » Test Environment Service (v3.4.0)","id":"1716","title":"Test Environment Service (v3.4.0)"},"1717":{"body":"A comprehensive containerized test environment service has been integrated into the orchestrator, enabling automated testing of taskservs, complete servers, and multi-node clusters without manual Docker management.","breadcrumbs":"Test Environment System » 🚀 Test Environment Service Completed (2025-10-06)","id":"1717","title":"🚀 Test Environment Service Completed (2025-10-06)"},"1718":{"body":"Automated Container Management : No manual Docker operations required Three Test Environment Types : Single taskserv, server simulation, multi-node clusters Multi-Node Support : Test complex topologies (Kubernetes HA, etcd clusters) Network Isolation : Each test environment gets dedicated Docker networks Resource Management : Configurable CPU, memory, and disk limits Topology Templates : Predefined cluster configurations for common scenarios Auto-Cleanup : Optional automatic cleanup after tests complete CI/CD Integration : Easy integration into automated pipelines","breadcrumbs":"Test Environment System » Key Features","id":"1718","title":"Key Features"},"1719":{"body":"","breadcrumbs":"Test Environment System » Test Environment Types","id":"1719","title":"Test Environment Types"},"172":{"body":"Deploy a complete cluster configuration: provisioning cluster create buildkit --infra my-infra","breadcrumbs":"First Deployment » Pattern 3: Complete Cluster","id":"172","title":"Pattern 3: Complete Cluster"},"1720":{"body":"Test individual taskserv in isolated container: # Quick test (create, run, cleanup)\\nprovisioning test quick kubernetes # With custom resources\\nprovisioning test env single postgres --cpu 2000 --memory 4096 --auto-start --auto-cleanup # With infrastructure context\\nprovisioning test env single redis --infra my-project\\n```plaintext ### 2. Server Simulation Test complete server configurations with multiple taskservs: ```bash\\n# Simulate web server\\nprovisioning test env server web-01 [containerd kubernetes cilium] --auto-start # Simulate database server\\nprovisioning test env server db-01 [postgres redis] --infra prod-stack --auto-start\\n```plaintext ### 3. Multi-Node Cluster Topology Test complex cluster configurations before deployment: ```bash\\n# 3-node Kubernetes HA cluster\\nprovisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start # etcd cluster\\nprovisioning test topology load etcd_cluster | test env cluster etcd --auto-start # Single-node Kubernetes\\nprovisioning test topology load kubernetes_single | test env cluster kubernetes\\n```plaintext ## Test Environment Management ```bash\\n# List all test environments\\nprovisioning test env list # Check environment status\\nprovisioning test env status  # View environment logs\\nprovisioning test env logs  # Run tests in environment\\nprovisioning test env run  # Cleanup environment\\nprovisioning test env cleanup \\n```plaintext ## Available Topology Templates Predefined multi-node cluster templates in `provisioning/config/test-topologies.toml`: | Template | Description | Nodes | Use Case |\\n|----------|-------------|-------|----------|\\n| `kubernetes_3node` | K8s HA cluster | 1 CP + 2 workers | Production-like testing |\\n| `kubernetes_single` | All-in-one K8s | 1 node | Development testing |\\n| `etcd_cluster` | etcd cluster | 3 members | Distributed consensus |\\n| `containerd_test` | Standalone containerd | 1 node | Container runtime |\\n| `postgres_redis` | Database stack | 2 nodes | Database integration | ## REST API Endpoints The orchestrator exposes test environment endpoints: - **Create Environment**: `POST http://localhost:9090/v1/test/environments/create`\\n- **List Environments**: `GET http://localhost:9090/v1/test/environments`\\n- **Get Environment**: `GET http://localhost:9090/v1/test/environments/{id}`\\n- **Run Tests**: `POST http://localhost:9090/v1/test/environments/{id}/run`\\n- **Cleanup**: `DELETE http://localhost:9090/v1/test/environments/{id}`\\n- **Get Logs**: `GET http://localhost:9090/v1/test/environments/{id}/logs` ## Prerequisites 1. **Docker Running**: Test environments require Docker daemon ```bash docker ps # Should work without errors Orchestrator Running : Start the orchestrator to manage test containers cd provisioning/platform/orchestrator\\n./scripts/start-orchestrator.nu --background","breadcrumbs":"Test Environment System » 1. Single Taskserv Testing","id":"1720","title":"1. Single Taskserv Testing"},"1721":{"body":"User Command (CLI/API) ↓\\nTest Orchestrator (Rust) ↓\\nContainer Manager (bollard) ↓\\nDocker API ↓\\nIsolated Test Containers • Dedicated networks • Resource limits • Volume mounts • Multi-node support\\n```plaintext ## Configuration - **Topology Templates**: `provisioning/config/test-topologies.toml`\\n- **Default Resources**: 1000 millicores CPU, 2048 MB memory\\n- **Network**: 172.20.0.0/16 (default subnet)\\n- **Base Image**: ubuntu:22.04 (configurable) ## Use Cases 1. **Taskserv Development**: Test new taskservs before deployment\\n2. **Integration Testing**: Validate taskserv combinations\\n3. **Cluster Validation**: Test multi-node configurations\\n4. **CI/CD Integration**: Automated infrastructure testing\\n5. **Production Simulation**: Test production-like deployments safely ## CI/CD Integration Example ```yaml\\n# GitLab CI\\ntest-infrastructure: stage: test script: - ./scripts/start-orchestrator.nu --background - provisioning test quick kubernetes - provisioning test quick postgres - provisioning test quick redis - provisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start artifacts: when: on_failure paths: - test-logs/\\n```plaintext ## Documentation Complete documentation available: - **User Guide**: [Test Environment Guide](../testing/test-environment-guide.md)\\n- **Detailed Usage**: [Test Environment Usage](../testing/test-environment-usage.md)\\n- **Orchestrator README**: [Orchestrator](../operations/orchestrator-system.md) ## Command Shortcuts Test commands are integrated into the CLI with shortcuts: - `test` or `tst` - Test command prefix\\n- `test quick ` - One-command test\\n- `test env single/server/cluster` - Create test environments\\n- `test topology load/list` - Manage topology templates","breadcrumbs":"Test Environment System » Architecture","id":"1721","title":"Architecture"},"1722":{"body":"Version : 1.0.0 Date : 2025-10-06 Status : Production Ready","breadcrumbs":"TaskServ Validation Guide » Taskserv Validation and Testing Guide","id":"1722","title":"Taskserv Validation and Testing Guide"},"1723":{"body":"The taskserv validation and testing system provides comprehensive evaluation of infrastructure services before deployment, reducing errors and increasing confidence in deployments.","breadcrumbs":"TaskServ Validation Guide » Overview","id":"1723","title":"Overview"},"1724":{"body":"","breadcrumbs":"TaskServ Validation Guide » Validation Levels","id":"1724","title":"Validation Levels"},"1725":{"body":"Validates configuration files, templates, and scripts without requiring infrastructure access. What it checks: KCL schema syntax and semantics Jinja2 template syntax Shell script syntax (with shellcheck if available) File structure and naming conventions Command: provisioning taskserv validate kubernetes --level static\\n```plaintext ### 2. Dependency Validation Checks taskserv dependencies, conflicts, and requirements. **What it checks:** - Required dependencies are available\\n- Optional dependencies status\\n- Conflicting taskservs\\n- Resource requirements (memory, CPU, disk)\\n- Health check configuration **Command:** ```bash\\nprovisioning taskserv validate kubernetes --level dependencies\\n```plaintext **Check against infrastructure:** ```bash\\nprovisioning taskserv check-deps kubernetes --infra my-project\\n```plaintext ### 3. Check Mode (Dry-Run) Enhanced check mode that performs validation and previews deployment without making changes. **What it does:** - Runs static validation\\n- Validates dependencies\\n- Previews configuration generation\\n- Lists files to be deployed\\n- Checks prerequisites (without SSH in check mode) **Command:** ```bash\\nprovisioning taskserv create kubernetes --check\\n```plaintext ### 4. Sandbox Testing Tests taskserv in isolated container environment before actual deployment. **What it tests:** - Package prerequisites\\n- Configuration validity\\n- Script execution\\n- Health check simulation **Command:** ```bash\\n# Test with Docker\\nprovisioning taskserv test kubernetes --runtime docker # Test with Podman\\nprovisioning taskserv test kubernetes --runtime podman # Keep container for inspection\\nprovisioning taskserv test kubernetes --runtime docker --keep\\n```plaintext --- ## Complete Validation Workflow ### Recommended Validation Sequence ```bash\\n# 1. Static validation (fastest, no infrastructure needed)\\nprovisioning taskserv validate kubernetes --level static -v # 2. Dependency validation\\nprovisioning taskserv check-deps kubernetes --infra my-project # 3. Check mode (dry-run with full validation)\\nprovisioning taskserv create kubernetes --check -v # 4. Sandbox testing (optional, requires Docker/Podman)\\nprovisioning taskserv test kubernetes --runtime docker # 5. Actual deployment (after all validations pass)\\nprovisioning taskserv create kubernetes\\n```plaintext ### Quick Validation (All Levels) ```bash\\n# Run all validation levels\\nprovisioning taskserv validate kubernetes --level all -v\\n```plaintext --- ## Validation Commands Reference ### `provisioning taskserv validate ` Multi-level validation framework. **Options:** - `--level ` - Validation level: static, dependencies, health, all (default: all)\\n- `--infra ` - Infrastructure context\\n- `--settings ` - Settings file path\\n- `--verbose` - Verbose output\\n- `--out ` - Output format: json, yaml, text **Examples:** ```bash\\n# Complete validation\\nprovisioning taskserv validate kubernetes # Only static validation\\nprovisioning taskserv validate kubernetes --level static # With verbose output\\nprovisioning taskserv validate kubernetes -v # JSON output\\nprovisioning taskserv validate kubernetes --out json\\n```plaintext ### `provisioning taskserv check-deps ` Check dependencies against infrastructure. **Options:** - `--infra ` - Infrastructure context\\n- `--settings ` - Settings file path\\n- `--verbose` - Verbose output **Examples:** ```bash\\n# Check dependencies\\nprovisioning taskserv check-deps kubernetes --infra my-project # Verbose output\\nprovisioning taskserv check-deps kubernetes --infra my-project -v\\n```plaintext ### `provisioning taskserv create  --check` Enhanced check mode with full validation and preview. **Options:** - `--check` - Enable check mode (no actual deployment)\\n- `--verbose` - Verbose output\\n- All standard create options **Examples:** ```bash\\n# Check mode with verbose output\\nprovisioning taskserv create kubernetes --check -v # Check specific server\\nprovisioning taskserv create kubernetes server-01 --check\\n```plaintext ### `provisioning taskserv test ` Sandbox testing in isolated environment. **Options:** - `--runtime ` - Runtime: docker, podman, native (default: docker)\\n- `--infra ` - Infrastructure context\\n- `--settings ` - Settings file path\\n- `--keep` - Keep container after test\\n- `--verbose` - Verbose output **Examples:** ```bash\\n# Test with Docker\\nprovisioning taskserv test kubernetes --runtime docker # Test with Podman\\nprovisioning taskserv test kubernetes --runtime podman # Keep container for debugging\\nprovisioning taskserv test kubernetes --keep -v # Connect to kept container\\ndocker exec -it taskserv-test-kubernetes bash\\n```plaintext --- ## Validation Output ### Static Validation ```plaintext\\nTaskserv Validation\\nTaskserv: kubernetes\\nLevel: static Validating KCL schemas for kubernetes... Checking kubernetes.k... ✓ Valid Checking version.k... ✓ Valid Checking dependencies.k... ✓ Valid Validating templates for kubernetes... Checking env-kubernetes.j2... ✓ Basic syntax OK Checking install-kubernetes.sh... ✓ Basic syntax OK Validation Summary\\n✓ kcl: 0 errors, 0 warnings\\n✓ templates: 0 errors, 0 warnings\\n✓ scripts: 0 errors, 0 warnings Overall Status\\n✓ VALID - 0 warnings\\n```plaintext ### Dependency Validation ```plaintext\\nDependency Validation Report\\nTaskserv: kubernetes Status: VALID Required Dependencies: • containerd • etcd • os Optional Dependencies: • cilium • helm Conflicts: • docker • podman\\n```plaintext ### Check Mode Output ```plaintext\\nCheck Mode: kubernetes on server-01 → Running static validation... ✓ Static validation passed → Checking dependencies... ✓ Dependencies OK Required: containerd, etcd, os → Previewing configuration generation... ✓ Configuration preview generated Files to process: 15 → Checking prerequisites... ℹ Prerequisite checks (preview mode): ⊘ Server accessibility: Check mode - SSH not tested ℹ Directory /tmp: Would verify directory exists ℹ Command bash: Would verify command is available Check Mode Summary\\n✓ All validations passed 💡 Taskserv can be deployed with: provisioning taskserv create kubernetes\\n```plaintext ### Test Output ```plaintext\\nTaskserv Sandbox Testing\\nTaskserv: kubernetes\\nRuntime: docker → Running pre-test validation...\\n✓ Validation passed → Preparing sandbox environment... Using base image: ubuntu:22.04\\n✓ Sandbox prepared: a1b2c3d4e5f6 → Running tests in sandbox... Test 1: Package prerequisites... Test 2: Configuration validity... Test 3: Script execution... Test 4: Health check simulation... Test Summary\\nTotal tests: 4\\nPassed: 4\\nFailed: 0\\nSkipped: 0 Detailed Results: ✓ Package prerequisites: Package manager accessible ✓ Configuration validity: 3 configuration files validated ✓ Script execution: 2 scripts validated ✓ Health check: Health check configuration valid: http://localhost:6443/healthz ✓ All tests passed\\n```plaintext --- ## Integration with CI/CD ### GitLab CI Example ```yaml\\nvalidate-taskservs: stage: validate script: - provisioning taskserv validate kubernetes --level all --out json - provisioning taskserv check-deps kubernetes --infra production test-taskservs: stage: test script: - provisioning taskserv test kubernetes --runtime docker dependencies: - validate-taskservs deploy-taskservs: stage: deploy script: - provisioning taskserv create kubernetes dependencies: - test-taskservs only: - main\\n```plaintext ### GitHub Actions Example ```yaml\\nname: Taskserv Validation on: [push, pull_request] jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Validate Taskservs run: | provisioning taskserv validate kubernetes --level all -v - name: Check Dependencies run: | provisioning taskserv check-deps kubernetes --infra production - name: Test in Sandbox run: | provisioning taskserv test kubernetes --runtime docker\\n```plaintext --- ## Troubleshooting ### shellcheck not found If shellcheck is not available, script validation will be skipped with a warning. **Install shellcheck:** ```bash\\n# macOS\\nbrew install shellcheck # Ubuntu/Debian\\napt install shellcheck # Fedora\\ndnf install shellcheck\\n```plaintext ### Docker/Podman not available Sandbox testing requires Docker or Podman. **Check runtime:** ```bash\\n# Docker\\ndocker ps # Podman\\npodman ps # Use native mode (limited testing)\\nprovisioning taskserv test kubernetes --runtime native\\n```plaintext ### KCL validation errors KCL schema errors indicate syntax or semantic problems. **Common fixes:** - Check schema syntax in `.k` files\\n- Validate imports and dependencies\\n- Run `kcl fmt` to format files\\n- Check `kcl.mod` dependencies ### Dependency conflicts If conflicting taskservs are detected: - Remove conflicting taskserv first\\n- Check infrastructure configuration\\n- Review dependency declarations in `dependencies.k` --- ## Advanced Usage ### Custom Validation Scripts You can create custom validation scripts by extending the validation framework: ```nushell\\n# custom_validation.nu\\nuse provisioning/core/nulib/taskservs/validate.nu * def custom-validate [taskserv: string] { # Custom validation logic let result = (validate-kcl-schemas $taskserv --verbose=true) # Additional custom checks # ... return $result\\n}\\n```plaintext ### Batch Validation Validate multiple taskservs: ```bash\\n# Validate all taskservs in infrastructure\\nfor taskserv in (provisioning taskserv list | get name) { provisioning taskserv validate $taskserv\\n}\\n```plaintext ### Automated Testing Create test suite for all taskservs: ```bash\\n#!/usr/bin/env nu let taskservs = [\\"kubernetes\\", \\"containerd\\", \\"cilium\\", \\"etcd\\"] for ts in $taskservs { print $\\"Testing ($ts)...\\" provisioning taskserv test $ts --runtime docker\\n}\\n```plaintext --- ## Best Practices ### Before Deployment 1. **Always validate** before deploying to production\\n2. **Run check mode** to preview changes\\n3. **Test in sandbox** for critical services\\n4. **Check dependencies** in infrastructure context ### During Development 1. **Validate frequently** during taskserv development\\n2. **Use verbose mode** to understand validation details\\n3. **Fix warnings** even if validation passes\\n4. **Keep containers** for debugging test failures ### In CI/CD 1. **Fail fast** on validation errors\\n2. **Require all tests pass** before merge\\n3. **Generate reports** in JSON format for analysis\\n4. **Archive test results** for audit trail --- ## Related Documentation - [Taskserv Development Guide](taskserv-development-guide.md)\\n- KCL Schema Reference\\n- [Dependency Management](dependency-management.md)\\n- [CI/CD Integration](cicd-integration.md) --- ## Version History | Version | Date | Changes |\\n|---------|------|---------|\\n| 1.0.0 | 2025-10-06 | Initial validation and testing guide | --- **Maintained By**: Infrastructure Team\\n**Review Cycle**: Quarterly","breadcrumbs":"TaskServ Validation Guide » 1. Static Validation","id":"1725","title":"1. Static Validation"},"1726":{"body":"This comprehensive troubleshooting guide helps you diagnose and resolve common issues with Infrastructure Automation.","breadcrumbs":"Troubleshooting Guide » Troubleshooting Guide","id":"1726","title":"Troubleshooting Guide"},"1727":{"body":"Common issues and their solutions Diagnostic commands and techniques Error message interpretation Performance optimization Recovery procedures Prevention strategies","breadcrumbs":"Troubleshooting Guide » What You\'ll Learn","id":"1727","title":"What You\'ll Learn"},"1728":{"body":"","breadcrumbs":"Troubleshooting Guide » General Troubleshooting Approach","id":"1728","title":"General Troubleshooting Approach"},"1729":{"body":"# Check overall system status\\nprovisioning env\\nprovisioning validate config # Check specific component status\\nprovisioning show servers --infra my-infra\\nprovisioning taskserv list --infra my-infra --installed\\n```plaintext ### 2. Gather Information ```bash\\n# Enable debug mode for detailed output\\nprovisioning --debug  # Check logs and errors\\nprovisioning show logs --infra my-infra\\n```plaintext ### 3. Use Diagnostic Commands ```bash\\n# Validate configuration\\nprovisioning validate config --detailed # Test connectivity\\nprovisioning provider test aws\\nprovisioning network test --infra my-infra\\n```plaintext ## Installation and Setup Issues ### Issue: Installation Fails **Symptoms:** - Installation script errors\\n- Missing dependencies\\n- Permission denied errors **Diagnosis:** ```bash\\n# Check system requirements\\nuname -a\\ndf -h\\nwhoami # Check permissions\\nls -la /usr/local/\\nsudo -l\\n```plaintext **Solutions:** #### Permission Issues ```bash\\n# Run installer with sudo\\nsudo ./install-provisioning # Or install to user directory\\n./install-provisioning --prefix=$HOME/provisioning\\nexport PATH=\\"$HOME/provisioning/bin:$PATH\\"\\n```plaintext #### Missing Dependencies ```bash\\n# Ubuntu/Debian\\nsudo apt update\\nsudo apt install -y curl wget tar build-essential # RHEL/CentOS\\nsudo dnf install -y curl wget tar gcc make\\n```plaintext #### Architecture Issues ```bash\\n# Check architecture\\nuname -m # Download correct architecture package\\n# x86_64: Intel/AMD 64-bit\\n# arm64: ARM 64-bit (Apple Silicon)\\nwget https://releases.example.com/provisioning-linux-x86_64.tar.gz\\n```plaintext ### Issue: Command Not Found **Symptoms:** ```plaintext\\nbash: provisioning: command not found\\n```plaintext **Diagnosis:** ```bash\\n# Check if provisioning is installed\\nwhich provisioning\\nls -la /usr/local/bin/provisioning # Check PATH\\necho $PATH\\n```plaintext **Solutions:** ```bash\\n# Add to PATH\\nexport PATH=\\"/usr/local/bin:$PATH\\" # Make permanent (add to shell profile)\\necho \'export PATH=\\"/usr/local/bin:$PATH\\"\' >> ~/.bashrc\\nsource ~/.bashrc # Create symlink if missing\\nsudo ln -sf /usr/local/provisioning/core/nulib/provisioning /usr/local/bin/provisioning\\n```plaintext ### Issue: Nushell Plugin Errors **Symptoms:** ```plaintext\\nPlugin not found: nu_plugin_kcl\\nPlugin registration failed\\n```plaintext **Diagnosis:** ```bash\\n# Check Nushell version\\nnu --version # Check KCL installation (required for nu_plugin_kcl)\\nkcl version # Check plugin registration\\nnu -c \\"version | get installed_plugins\\"\\n```plaintext **Solutions:** ```bash\\n# Install KCL CLI (required for nu_plugin_kcl)\\n# Download from: https://github.com/kcl-lang/cli/releases # Re-register plugins\\nnu -c \\"plugin add /usr/local/provisioning/plugins/nu_plugin_kcl\\"\\nnu -c \\"plugin add /usr/local/provisioning/plugins/nu_plugin_tera\\" # Restart Nushell after plugin registration\\n```plaintext ## Configuration Issues ### Issue: Configuration Not Found **Symptoms:** ```plaintext\\nConfiguration file not found\\nFailed to load configuration\\n```plaintext **Diagnosis:** ```bash\\n# Check configuration file locations\\nprovisioning env | grep config # Check if files exist\\nls -la ~/.config/provisioning/\\nls -la /usr/local/provisioning/config.defaults.toml\\n```plaintext **Solutions:** ```bash\\n# Initialize user configuration\\nprovisioning init config # Create missing directories\\nmkdir -p ~/.config/provisioning # Copy template\\ncp /usr/local/provisioning/config-examples/config.user.toml ~/.config/provisioning/config.toml # Verify configuration\\nprovisioning validate config\\n```plaintext ### Issue: Configuration Validation Errors **Symptoms:** ```plaintext\\nConfiguration validation failed\\nInvalid configuration value\\nMissing required field\\n```plaintext **Diagnosis:** ```bash\\n# Detailed validation\\nprovisioning validate config --detailed # Check specific sections\\nprovisioning config show --section paths\\nprovisioning config show --section providers\\n```plaintext **Solutions:** #### Path Configuration Issues ```bash\\n# Check base path exists\\nls -la /path/to/provisioning # Update configuration\\nnano ~/.config/provisioning/config.toml # Fix paths section\\n[paths]\\nbase = \\"/correct/path/to/provisioning\\"\\n```plaintext #### Provider Configuration Issues ```bash\\n# Test provider connectivity\\nprovisioning provider test aws # Check credentials\\naws configure list # For AWS\\nupcloud-cli config # For UpCloud # Update provider configuration\\n[providers.aws]\\ninterface = \\"CLI\\" # or \\"API\\"\\n```plaintext ### Issue: Interpolation Failures **Symptoms:** ```plaintext\\nInterpolation pattern not resolved: {{env.VARIABLE}}\\nTemplate rendering failed\\n```plaintext **Diagnosis:** ```bash\\n# Test interpolation\\nprovisioning validate interpolation test # Check environment variables\\nenv | grep VARIABLE # Debug interpolation\\nprovisioning --debug validate interpolation validate\\n```plaintext **Solutions:** ```bash\\n# Set missing environment variables\\nexport MISSING_VARIABLE=\\"value\\" # Use fallback values in configuration\\nconfig_value = \\"{{env.VARIABLE || \'default_value\'}}\\" # Check interpolation syntax\\n# Correct: {{env.HOME}}\\n# Incorrect: ${HOME} or $HOME\\n```plaintext ## Server Management Issues ### Issue: Server Creation Fails **Symptoms:** ```plaintext\\nFailed to create server\\nProvider API error\\nInsufficient quota\\n```plaintext **Diagnosis:** ```bash\\n# Check provider status\\nprovisioning provider status aws # Test connectivity\\nping api.provider.com\\ncurl -I https://api.provider.com # Check quota\\nprovisioning provider quota --infra my-infra # Debug server creation\\nprovisioning --debug server create web-01 --infra my-infra --check\\n```plaintext **Solutions:** #### API Authentication Issues ```bash\\n# AWS\\naws configure list\\naws sts get-caller-identity # UpCloud\\nupcloud-cli account show # Update credentials\\naws configure # For AWS\\nexport UPCLOUD_USERNAME=\\"your-username\\"\\nexport UPCLOUD_PASSWORD=\\"your-password\\"\\n```plaintext #### Quota/Limit Issues ```bash\\n# Check current usage\\nprovisioning show costs --infra my-infra # Request quota increase from provider\\n# Or reduce resource requirements # Use smaller instance types\\n# Reduce number of servers\\n```plaintext #### Network/Connectivity Issues ```bash\\n# Test network connectivity\\ncurl -v https://api.aws.amazon.com\\ncurl -v https://api.upcloud.com # Check DNS resolution\\nnslookup api.aws.amazon.com # Check firewall rules\\n# Ensure outbound HTTPS (port 443) is allowed\\n```plaintext ### Issue: SSH Access Fails **Symptoms:** ```plaintext\\nConnection refused\\nPermission denied\\nHost key verification failed\\n```plaintext **Diagnosis:** ```bash\\n# Check server status\\nprovisioning server list --infra my-infra # Test SSH manually\\nssh -v user@server-ip # Check SSH configuration\\nprovisioning show servers web-01 --infra my-infra\\n```plaintext **Solutions:** #### Connection Issues ```bash\\n# Wait for server to be fully ready\\nprovisioning server list --infra my-infra --status # Check security groups/firewall\\n# Ensure SSH (port 22) is allowed # Use correct IP address\\nprovisioning show servers web-01 --infra my-infra | grep ip\\n```plaintext #### Authentication Issues ```bash\\n# Check SSH key\\nls -la ~/.ssh/\\nssh-add -l # Generate new key if needed\\nssh-keygen -t ed25519 -f ~/.ssh/provisioning_key # Use specific key\\nprovisioning server ssh web-01 --key ~/.ssh/provisioning_key --infra my-infra\\n```plaintext #### Host Key Issues ```bash\\n# Remove old host key\\nssh-keygen -R server-ip # Accept new host key\\nssh -o StrictHostKeyChecking=accept-new user@server-ip\\n```plaintext ## Task Service Issues ### Issue: Service Installation Fails **Symptoms:** ```plaintext\\nService installation failed\\nPackage not found\\nDependency conflicts\\n```plaintext **Diagnosis:** ```bash\\n# Check service prerequisites\\nprovisioning taskserv check kubernetes --infra my-infra # Debug installation\\nprovisioning --debug taskserv create kubernetes --infra my-infra --check # Check server resources\\nprovisioning server ssh web-01 --command \\"free -h && df -h\\" --infra my-infra\\n```plaintext **Solutions:** #### Resource Issues ```bash\\n# Check available resources\\nprovisioning server ssh web-01 --command \\" echo \'Memory:\' && free -h echo \'Disk:\' && df -h echo \'CPU:\' && nproc\\n\\" --infra my-infra # Upgrade server if needed\\nprovisioning server resize web-01 --plan larger-plan --infra my-infra\\n```plaintext #### Package Repository Issues ```bash\\n# Update package lists\\nprovisioning server ssh web-01 --command \\" sudo apt update && sudo apt upgrade -y\\n\\" --infra my-infra # Check repository connectivity\\nprovisioning server ssh web-01 --command \\" curl -I https://download.docker.com/linux/ubuntu/\\n\\" --infra my-infra\\n```plaintext #### Dependency Issues ```bash\\n# Install missing dependencies\\nprovisioning taskserv create containerd --infra my-infra # Then install dependent service\\nprovisioning taskserv create kubernetes --infra my-infra\\n```plaintext ### Issue: Service Not Running **Symptoms:** ```plaintext\\nService status: failed\\nService not responding\\nHealth check failures\\n```plaintext **Diagnosis:** ```bash\\n# Check service status\\nprovisioning taskserv status kubernetes --infra my-infra # Check service logs\\nprovisioning taskserv logs kubernetes --infra my-infra # SSH and check manually\\nprovisioning server ssh web-01 --command \\" sudo systemctl status kubernetes sudo journalctl -u kubernetes --no-pager -n 50\\n\\" --infra my-infra\\n```plaintext **Solutions:** #### Configuration Issues ```bash\\n# Reconfigure service\\nprovisioning taskserv configure kubernetes --infra my-infra # Reset to defaults\\nprovisioning taskserv reset kubernetes --infra my-infra\\n```plaintext #### Port Conflicts ```bash\\n# Check port usage\\nprovisioning server ssh web-01 --command \\" sudo netstat -tulpn | grep :6443 sudo ss -tulpn | grep :6443\\n\\" --infra my-infra # Change port configuration or stop conflicting service\\n```plaintext #### Permission Issues ```bash\\n# Fix permissions\\nprovisioning server ssh web-01 --command \\" sudo chown -R kubernetes:kubernetes /var/lib/kubernetes sudo chmod 600 /etc/kubernetes/admin.conf\\n\\" --infra my-infra\\n```plaintext ## Cluster Management Issues ### Issue: Cluster Deployment Fails **Symptoms:** ```plaintext\\nCluster deployment failed\\nPod creation errors\\nService unavailable\\n```plaintext **Diagnosis:** ```bash\\n# Check cluster status\\nprovisioning cluster status web-cluster --infra my-infra # Check Kubernetes cluster\\nprovisioning server ssh master-01 --command \\" kubectl get nodes kubectl get pods --all-namespaces\\n\\" --infra my-infra # Check cluster logs\\nprovisioning cluster logs web-cluster --infra my-infra\\n```plaintext **Solutions:** #### Node Issues ```bash\\n# Check node status\\nprovisioning server ssh master-01 --command \\" kubectl describe nodes\\n\\" --infra my-infra # Drain and rejoin problematic nodes\\nprovisioning server ssh master-01 --command \\" kubectl drain worker-01 --ignore-daemonsets kubectl delete node worker-01\\n\\" --infra my-infra # Rejoin node\\nprovisioning taskserv configure kubernetes --infra my-infra --servers worker-01\\n```plaintext #### Resource Constraints ```bash\\n# Check resource usage\\nprovisioning server ssh master-01 --command \\" kubectl top nodes kubectl top pods --all-namespaces\\n\\" --infra my-infra # Scale down or add more nodes\\nprovisioning cluster scale web-cluster --replicas 3 --infra my-infra\\nprovisioning server create worker-04 --infra my-infra\\n```plaintext #### Network Issues ```bash\\n# Check network plugin\\nprovisioning server ssh master-01 --command \\" kubectl get pods -n kube-system | grep cilium\\n\\" --infra my-infra # Restart network plugin\\nprovisioning taskserv restart cilium --infra my-infra\\n```plaintext ## Performance Issues ### Issue: Slow Operations **Symptoms:** - Commands take very long to complete\\n- Timeouts during operations\\n- High CPU/memory usage **Diagnosis:** ```bash\\n# Check system resources\\ntop\\nhtop\\nfree -h\\ndf -h # Check network latency\\nping api.aws.amazon.com\\ntraceroute api.aws.amazon.com # Profile command execution\\ntime provisioning server list --infra my-infra\\n```plaintext **Solutions:** #### Local System Issues ```bash\\n# Close unnecessary applications\\n# Upgrade system resources\\n# Use SSD storage if available # Increase timeout values\\nexport PROVISIONING_TIMEOUT=600 # 10 minutes\\n```plaintext #### Network Issues ```bash\\n# Use region closer to your location\\n[providers.aws]\\nregion = \\"us-west-1\\" # Closer region # Enable connection pooling/caching\\n[cache]\\nenabled = true\\n```plaintext #### Large Infrastructure Issues ```bash\\n# Use parallel operations\\nprovisioning server create --infra my-infra --parallel 4 # Filter results\\nprovisioning server list --infra my-infra --filter \\"status == \'running\'\\"\\n```plaintext ### Issue: High Memory Usage **Symptoms:** - System becomes unresponsive\\n- Out of memory errors\\n- Swap usage high **Diagnosis:** ```bash\\n# Check memory usage\\nfree -h\\nps aux --sort=-%mem | head # Check for memory leaks\\nvalgrind provisioning server list --infra my-infra\\n```plaintext **Solutions:** ```bash\\n# Increase system memory\\n# Close other applications\\n# Use streaming operations for large datasets # Enable garbage collection\\nexport PROVISIONING_GC_ENABLED=true # Reduce concurrent operations\\nexport PROVISIONING_MAX_PARALLEL=2\\n```plaintext ## Network and Connectivity Issues ### Issue: API Connectivity Problems **Symptoms:** ```plaintext\\nConnection timeout\\nDNS resolution failed\\nSSL certificate errors\\n```plaintext **Diagnosis:** ```bash\\n# Test basic connectivity\\nping 8.8.8.8\\ncurl -I https://api.aws.amazon.com\\nnslookup api.upcloud.com # Check SSL certificates\\nopenssl s_client -connect api.aws.amazon.com:443 -servername api.aws.amazon.com\\n```plaintext **Solutions:** #### DNS Issues ```bash\\n# Use alternative DNS\\necho \'nameserver 8.8.8.8\' | sudo tee /etc/resolv.conf # Clear DNS cache\\nsudo systemctl restart systemd-resolved # Ubuntu\\nsudo dscacheutil -flushcache # macOS\\n```plaintext #### Proxy/Firewall Issues ```bash\\n# Configure proxy if needed\\nexport HTTP_PROXY=http://proxy.company.com:9090\\nexport HTTPS_PROXY=http://proxy.company.com:9090 # Check firewall rules\\nsudo ufw status # Ubuntu\\nsudo firewall-cmd --list-all # RHEL/CentOS\\n```plaintext #### Certificate Issues ```bash\\n# Update CA certificates\\nsudo apt update && sudo apt install ca-certificates # Ubuntu\\nbrew install ca-certificates # macOS # Skip SSL verification (temporary)\\nexport PROVISIONING_SKIP_SSL_VERIFY=true\\n```plaintext ## Security and Encryption Issues ### Issue: SOPS Decryption Fails **Symptoms:** ```plaintext\\nSOPS decryption failed\\nAge key not found\\nInvalid key format\\n```plaintext **Diagnosis:** ```bash\\n# Check SOPS configuration\\nprovisioning sops config # Test SOPS manually\\nsops -d encrypted-file.k # Check Age keys\\nls -la ~/.config/sops/age/keys.txt\\nage-keygen -y ~/.config/sops/age/keys.txt\\n```plaintext **Solutions:** #### Missing Keys ```bash\\n# Generate new Age key\\nage-keygen -o ~/.config/sops/age/keys.txt # Update SOPS configuration\\nprovisioning sops config --key-file ~/.config/sops/age/keys.txt\\n```plaintext #### Key Permissions ```bash\\n# Fix key file permissions\\nchmod 600 ~/.config/sops/age/keys.txt\\nchown $(whoami) ~/.config/sops/age/keys.txt\\n```plaintext #### Configuration Issues ```bash\\n# Update SOPS configuration in ~/.config/provisioning/config.toml\\n[sops]\\nuse_sops = true\\nkey_search_paths = [ \\"~/.config/sops/age/keys.txt\\", \\"/path/to/your/key.txt\\"\\n]\\n```plaintext ### Issue: Access Denied Errors **Symptoms:** ```plaintext\\nPermission denied\\nAccess denied\\nInsufficient privileges\\n```plaintext **Diagnosis:** ```bash\\n# Check user permissions\\nid\\ngroups # Check file permissions\\nls -la ~/.config/provisioning/\\nls -la /usr/local/provisioning/ # Test with sudo\\nsudo provisioning env\\n```plaintext **Solutions:** ```bash\\n# Fix file ownership\\nsudo chown -R $(whoami):$(whoami) ~/.config/provisioning/ # Fix permissions\\nchmod -R 755 ~/.config/provisioning/\\nchmod 600 ~/.config/provisioning/config.toml # Add user to required groups\\nsudo usermod -a -G docker $(whoami) # For Docker access\\n```plaintext ## Data and Storage Issues ### Issue: Disk Space Problems **Symptoms:** ```plaintext\\nNo space left on device\\nWrite failed\\nDisk full\\n```plaintext **Diagnosis:** ```bash\\n# Check disk usage\\ndf -h\\ndu -sh ~/.config/provisioning/\\ndu -sh /usr/local/provisioning/ # Find large files\\nfind /usr/local/provisioning -type f -size +100M\\n```plaintext **Solutions:** ```bash\\n# Clean up cache files\\nrm -rf ~/.config/provisioning/cache/*\\nrm -rf /usr/local/provisioning/.cache/* # Clean up logs\\nfind /usr/local/provisioning -name \\"*.log\\" -mtime +30 -delete # Clean up temporary files\\nrm -rf /tmp/provisioning-* # Compress old backups\\ngzip ~/.config/provisioning/backups/*.yaml\\n```plaintext ## Recovery Procedures ### Configuration Recovery ```bash\\n# Restore from backup\\nprovisioning config restore --backup latest # Reset to defaults\\nprovisioning config reset # Recreate configuration\\nprovisioning init config --force\\n```plaintext ### Infrastructure Recovery ```bash\\n# Check infrastructure status\\nprovisioning show servers --infra my-infra # Recover failed servers\\nprovisioning server create failed-server --infra my-infra # Restore from backup\\nprovisioning restore --backup latest --infra my-infra\\n```plaintext ### Service Recovery ```bash\\n# Restart failed services\\nprovisioning taskserv restart kubernetes --infra my-infra # Reinstall corrupted services\\nprovisioning taskserv delete kubernetes --infra my-infra\\nprovisioning taskserv create kubernetes --infra my-infra\\n```plaintext ## Prevention Strategies ### Regular Maintenance ```bash\\n# Weekly maintenance script\\n#!/bin/bash # Update system\\nprovisioning update --check # Validate configuration\\nprovisioning validate config # Check for service updates\\nprovisioning taskserv check-updates # Clean up old files\\nprovisioning cleanup --older-than 30d # Create backup\\nprovisioning backup create --name \\"weekly-$(date +%Y%m%d)\\"\\n```plaintext ### Monitoring Setup ```bash\\n# Set up health monitoring\\n#!/bin/bash # Check system health every hour\\n0 * * * * /usr/local/bin/provisioning health check || echo \\"Health check failed\\" | mail -s \\"Provisioning Alert\\" admin@company.com # Weekly cost reports\\n0 9 * * 1 /usr/local/bin/provisioning show costs --all | mail -s \\"Weekly Cost Report\\" finance@company.com\\n```plaintext ### Best Practices 1. **Configuration Management** - Version control all configuration files - Use check mode before applying changes - Regular validation and testing 2. **Security** - Regular key rotation - Principle of least privilege - Audit logs review 3. **Backup Strategy** - Automated daily backups - Test restore procedures - Off-site backup storage 4. **Documentation** - Document custom configurations - Keep troubleshooting logs - Share knowledge with team ## Getting Additional Help ### Debug Information Collection ```bash\\n#!/bin/bash\\n# Collect debug information echo \\"Collecting provisioning debug information...\\" mkdir -p /tmp/provisioning-debug\\ncd /tmp/provisioning-debug # System information\\nuname -a > system-info.txt\\nfree -h >> system-info.txt\\ndf -h >> system-info.txt # Provisioning information\\nprovisioning --version > provisioning-info.txt\\nprovisioning env >> provisioning-info.txt\\nprovisioning validate config --detailed > config-validation.txt 2>&1 # Configuration files\\ncp ~/.config/provisioning/config.toml user-config.toml 2>/dev/null || echo \\"No user config\\" > user-config.toml # Logs\\nprovisioning show logs > system-logs.txt 2>&1 # Create archive\\ncd /tmp\\ntar czf provisioning-debug-$(date +%Y%m%d_%H%M%S).tar.gz provisioning-debug/ echo \\"Debug information collected in: provisioning-debug-*.tar.gz\\"\\n```plaintext ### Support Channels 1. **Built-in Help** ```bash provisioning help provisioning help  Documentation User guides in docs/user/ CLI reference: docs/user/cli-reference.md Configuration guide: docs/user/configuration.md Community Resources Project repository issues Community forums Documentation wiki Enterprise Support Professional services Priority support Custom development Remember: When reporting issues, always include the debug information collected above and specific error messages.","breadcrumbs":"Troubleshooting Guide » 1. Identify the Problem","id":"1729","title":"1. Identify the Problem"},"173":{"body":"The typical deployment workflow: # 1. Initialize workspace\\nprovisioning workspace init production # 2. Generate infrastructure\\nprovisioning generate infra --new prod-infra # 3. Configure (edit settings.k)\\n$EDITOR workspace/infra/prod-infra/settings.k # 4. Validate configuration\\nprovisioning validate config --infra prod-infra # 5. Create servers (check mode)\\nprovisioning server create --infra prod-infra --check # 6. Create servers (real)\\nprovisioning server create --infra prod-infra # 7. Install task services\\nprovisioning taskserv create kubernetes --infra prod-infra --wait # 8. Deploy cluster (if needed)\\nprovisioning cluster create my-cluster --infra prod-infra # 9. Verify\\nprovisioning server list\\nprovisioning taskserv list","breadcrumbs":"First Deployment » Deployment Workflow","id":"173","title":"Deployment Workflow"},"1730":{"body":"Version : 3.5.0 Last Updated : 2025-10-09 Estimated Time : 30-60 minutes Difficulty : Beginner to Intermediate","breadcrumbs":"From Scratch » Complete Deployment Guide: From Scratch to Production","id":"1730","title":"Complete Deployment Guide: From Scratch to Production"},"1731":{"body":"Prerequisites Step 1: Install Nushell Step 2: Install Nushell Plugins (Recommended) Step 3: Install Required Tools Step 4: Clone and Setup Project Step 5: Initialize Workspace Step 6: Configure Environment Step 7: Discover and Load Modules Step 8: Validate Configuration Step 9: Deploy Servers Step 10: Install Task Services Step 11: Create Clusters Step 12: Verify Deployment Step 13: Post-Deployment Troubleshooting Next Steps","breadcrumbs":"From Scratch » Table of Contents","id":"1731","title":"Table of Contents"},"1732":{"body":"Before starting, ensure you have: ✅ Operating System : macOS, Linux, or Windows (WSL2 recommended) ✅ Administrator Access : Ability to install software and configure system ✅ Internet Connection : For downloading dependencies and accessing cloud providers ✅ Cloud Provider Credentials : UpCloud, AWS, or local development environment ✅ Basic Terminal Knowledge : Comfortable running shell commands ✅ Text Editor : vim, nano, VSCode, or your preferred editor","breadcrumbs":"From Scratch » Prerequisites","id":"1732","title":"Prerequisites"},"1733":{"body":"CPU : 2+ cores RAM : 8GB minimum, 16GB recommended Disk : 20GB free space minimum","breadcrumbs":"From Scratch » Recommended Hardware","id":"1733","title":"Recommended Hardware"},"1734":{"body":"Nushell 0.107.1+ is the primary shell and scripting language for the provisioning platform.","breadcrumbs":"From Scratch » Step 1: Install Nushell","id":"1734","title":"Step 1: Install Nushell"},"1735":{"body":"# Install Nushell\\nbrew install nushell # Verify installation\\nnu --version\\n# Expected: 0.107.1 or higher\\n```plaintext ### Linux (via Package Manager) **Ubuntu/Debian:** ```bash\\n# Add Nushell repository\\ncurl -fsSL https://starship.rs/install.sh | bash # Install Nushell\\nsudo apt update\\nsudo apt install nushell # Verify installation\\nnu --version\\n```plaintext **Fedora:** ```bash\\nsudo dnf install nushell\\nnu --version\\n```plaintext **Arch Linux:** ```bash\\nsudo pacman -S nushell\\nnu --version\\n```plaintext ### Linux/macOS (via Cargo) ```bash\\n# Install Rust (if not already installed)\\ncurl --proto \'=https\' --tlsv1.2 -sSf https://sh.rustup.rs | sh\\nsource $HOME/.cargo/env # Install Nushell\\ncargo install nu --locked # Verify installation\\nnu --version\\n```plaintext ### Windows (via Winget) ```powershell\\n# Install Nushell\\nwinget install nushell # Verify installation\\nnu --version\\n```plaintext ### Configure Nushell ```bash\\n# Start Nushell\\nnu # Configure (creates default config if not exists)\\nconfig nu\\n```plaintext --- ## Step 2: Install Nushell Plugins (Recommended) Native plugins provide **10-50x performance improvement** for authentication, KMS, and orchestrator operations. ### Why Install Plugins? **Performance Gains:** - 🚀 **KMS operations**: ~5ms vs ~50ms (10x faster)\\n- 🚀 **Orchestrator queries**: ~1ms vs ~30ms (30x faster)\\n- 🚀 **Batch encryption**: 100 files in 0.5s vs 5s (10x faster) **Benefits:** - ✅ Native Nushell integration (pipelines, data structures)\\n- ✅ OS keyring for secure token storage\\n- ✅ Offline capability (Age encryption, local orchestrator)\\n- ✅ Graceful fallback to HTTP if not installed ### Prerequisites for Building Plugins ```bash\\n# Install Rust toolchain (if not already installed)\\ncurl --proto \'=https\' --tlsv1.2 -sSf https://sh.rustup.rs | sh\\nsource $HOME/.cargo/env\\nrustc --version\\n# Expected: rustc 1.75+ or higher # Linux only: Install development packages\\nsudo apt install libssl-dev pkg-config # Ubuntu/Debian\\nsudo dnf install openssl-devel # Fedora # Linux only: Install keyring service (required for auth plugin)\\nsudo apt install gnome-keyring # Ubuntu/Debian (GNOME)\\nsudo apt install kwalletmanager # Ubuntu/Debian (KDE)\\n```plaintext ### Build Plugins ```bash\\n# Navigate to plugins directory\\ncd provisioning/core/plugins/nushell-plugins # Build all three plugins in release mode (optimized)\\ncargo build --release --all # Expected output:\\n# Compiling nu_plugin_auth v0.1.0\\n# Compiling nu_plugin_kms v0.1.0\\n# Compiling nu_plugin_orchestrator v0.1.0\\n# Finished release [optimized] target(s) in 2m 15s\\n```plaintext **Build time**: ~2-5 minutes depending on hardware ### Register Plugins with Nushell ```bash\\n# Register all three plugins (full paths recommended)\\nplugin add $PWD/target/release/nu_plugin_auth\\nplugin add $PWD/target/release/nu_plugin_kms\\nplugin add $PWD/target/release/nu_plugin_orchestrator # Alternative (from plugins directory)\\nplugin add target/release/nu_plugin_auth\\nplugin add target/release/nu_plugin_kms\\nplugin add target/release/nu_plugin_orchestrator\\n```plaintext ### Verify Plugin Installation ```bash\\n# List registered plugins\\nplugin list | where name =~ \\"auth|kms|orch\\" # Expected output:\\n# ╭───┬─────────────────────────┬─────────┬───────────────────────────────────╮\\n# │ # │ name │ version │ filename │\\n# ├───┼─────────────────────────┼─────────┼───────────────────────────────────┤\\n# │ 0 │ nu_plugin_auth │ 0.1.0 │ .../nu_plugin_auth │\\n# │ 1 │ nu_plugin_kms │ 0.1.0 │ .../nu_plugin_kms │\\n# │ 2 │ nu_plugin_orchestrator │ 0.1.0 │ .../nu_plugin_orchestrator │\\n# ╰───┴─────────────────────────┴─────────┴───────────────────────────────────╯ # Test each plugin\\nauth --help # Should show auth commands\\nkms --help # Should show kms commands\\norch --help # Should show orch commands\\n```plaintext ### Configure Plugin Environments ```bash\\n# Add to ~/.config/nushell/env.nu\\n$env.CONTROL_CENTER_URL = \\"http://localhost:3000\\"\\n$env.RUSTYVAULT_ADDR = \\"http://localhost:8200\\"\\n$env.RUSTYVAULT_TOKEN = \\"your-vault-token-here\\"\\n$env.ORCHESTRATOR_DATA_DIR = \\"provisioning/platform/orchestrator/data\\" # For Age encryption (local development)\\n$env.AGE_IDENTITY = $\\"($env.HOME)/.age/key.txt\\"\\n$env.AGE_RECIPIENT = \\"age1xxxxxxxxx\\" # Replace with your public key\\n```plaintext ### Test Plugins (Quick Smoke Test) ```bash\\n# Test KMS plugin (requires backend configured)\\nkms status\\n# Expected: { backend: \\"rustyvault\\", status: \\"healthy\\", ... }\\n# Or: Error if backend not configured (OK for now) # Test orchestrator plugin (reads local files)\\norch status\\n# Expected: { active_tasks: 0, completed_tasks: 0, health: \\"healthy\\" }\\n# Or: Error if orchestrator not started yet (OK for now) # Test auth plugin (requires control center)\\nauth verify\\n# Expected: { active: false }\\n# Or: Error if control center not running (OK for now)\\n```plaintext **Note**: It\'s OK if plugins show errors at this stage. We\'ll configure backends and services later. ### Skip Plugins? (Not Recommended) If you want to skip plugin installation for now: - ✅ All features work via HTTP API (slower but functional)\\n- ⚠️ You\'ll miss 10-50x performance improvements\\n- ⚠️ No offline capability for KMS/orchestrator\\n- ℹ️ You can install plugins later anytime To use HTTP fallback: ```bash\\n# System automatically uses HTTP if plugins not available\\n# No configuration changes needed\\n```plaintext --- ## Step 3: Install Required Tools ### Essential Tools **KCL (Configuration Language)** ```bash\\n# macOS\\nbrew install kcl # Linux\\ncurl -fsSL https://kcl-lang.io/script/install.sh | /bin/bash # Verify\\nkcl version\\n# Expected: 0.11.2 or higher\\n```plaintext **SOPS (Secrets Management)** ```bash\\n# macOS\\nbrew install sops # Linux\\nwget https://github.com/mozilla/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64\\nsudo mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops\\nsudo chmod +x /usr/local/bin/sops # Verify\\nsops --version\\n# Expected: 3.10.2 or higher\\n```plaintext **Age (Encryption Tool)** ```bash\\n# macOS\\nbrew install age # Linux\\nsudo apt install age # Ubuntu/Debian\\nsudo dnf install age # Fedora # Or from source\\ngo install filippo.io/age/cmd/...@latest # Verify\\nage --version\\n# Expected: 1.2.1 or higher # Generate Age key (for local encryption)\\nage-keygen -o ~/.age/key.txt\\ncat ~/.age/key.txt\\n# Save the public key (age1...) for later\\n```plaintext ### Optional but Recommended Tools **K9s (Kubernetes Management)** ```bash\\n# macOS\\nbrew install k9s # Linux\\ncurl -sS https://webinstall.dev/k9s | bash # Verify\\nk9s version\\n# Expected: 0.50.6 or higher\\n```plaintext **glow (Markdown Renderer)** ```bash\\n# macOS\\nbrew install glow # Linux\\nsudo apt install glow # Ubuntu/Debian\\nsudo dnf install glow # Fedora # Verify\\nglow --version\\n```plaintext --- ## Step 4: Clone and Setup Project ### Clone Repository ```bash\\n# Clone project\\ngit clone https://github.com/your-org/project-provisioning.git\\ncd project-provisioning # Or if already cloned, update to latest\\ngit pull origin main\\n```plaintext ### Add CLI to PATH (Optional) ```bash\\n# Add to ~/.bashrc or ~/.zshrc\\nexport PATH=\\"$PATH:/Users/Akasha/project-provisioning/provisioning/core/cli\\" # Or create symlink\\nsudo ln -s /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning /usr/local/bin/provisioning # Verify\\nprovisioning version\\n# Expected: 3.5.0\\n```plaintext --- ## Step 5: Initialize Workspace A workspace is a self-contained environment for managing infrastructure. ### Create New Workspace ```bash\\n# Initialize new workspace\\nprovisioning workspace init --name production # Or use interactive mode\\nprovisioning workspace init\\n# Name: production\\n# Description: Production infrastructure\\n# Provider: upcloud\\n```plaintext **What this creates:** The new workspace initialization now generates **KCL (Kusion Configuration Language) configuration files** for type-safe, schema-validated infrastructure definitions: ```plaintext\\nworkspace/\\n├── config/\\n│ ├── provisioning.k # Main KCL configuration (schema-validated)\\n│ ├── providers/\\n│ │ └── upcloud.toml # Provider-specific settings\\n│ ├── platform/ # Platform service configs\\n│ └── kms.toml # Key management settings\\n├── infra/ # Infrastructure definitions\\n├── extensions/ # Custom modules\\n└── runtime/ # Runtime data and state\\n```plaintext ### Workspace Configuration Format The workspace configuration now uses **KCL (type-safe)** instead of YAML. This provides: - ✅ **Type Safety**: Schema validation catches errors at load time\\n- ✅ **Immutability**: Enforces configuration immutability by default\\n- ✅ **Validation**: Semantic versioning, required fields, value constraints\\n- ✅ **Documentation**: Self-documenting with schema descriptions **Example KCL config** (`provisioning.k`): ```kcl\\nimport provisioning.workspace_config as ws workspace_config = ws.WorkspaceConfig { workspace: { name: \\"production\\" version: \\"1.0.0\\" created: \\"2025-12-03T14:30:00Z\\" } paths: { base: \\"/opt/workspaces/production\\" infra: \\"/opt/workspaces/production/infra\\" cache: \\"/opt/workspaces/production/.cache\\" # ... other paths } providers: { active: [\\"upcloud\\"] default: \\"upcloud\\" } # ... other sections\\n}\\n```plaintext **Backward Compatibility**: If you have existing YAML workspace configs (`provisioning.yaml`), they continue to work. The config loader checks for KCL files first, then falls back to YAML. ### Verify Workspace ```bash\\n# Show workspace info\\nprovisioning workspace info # List all workspaces\\nprovisioning workspace list # Show active workspace\\nprovisioning workspace active\\n# Expected: production\\n```plaintext ### View and Validate Workspace Configuration Now you can inspect and validate your KCL workspace configuration: ```bash\\n# View complete workspace configuration\\nprovisioning workspace config show # Show specific workspace\\nprovisioning workspace config show production # View configuration in different formats\\nprovisioning workspace config show --format=json\\nprovisioning workspace config show --format=yaml\\nprovisioning workspace config show --format=kcl # Raw KCL file # Validate workspace configuration\\nprovisioning workspace config validate\\n# Output: ✅ Validation complete - all configs are valid # Show configuration hierarchy (priority order)\\nprovisioning workspace config hierarchy\\n```plaintext **Configuration Validation**: The KCL schema automatically validates: - ✅ Semantic versioning format (e.g., \\"1.0.0\\")\\n- ✅ Required sections present (workspace, paths, provisioning, etc.)\\n- ✅ Valid file paths and types\\n- ✅ Provider configuration exists for active providers\\n- ✅ KMS and SOPS settings properly configured --- ## Step 6: Configure Environment ### Set Provider Credentials **UpCloud Provider:** ```bash\\n# Create provider config\\nvim workspace/config/providers/upcloud.toml\\n```plaintext ```toml\\n[upcloud]\\nusername = \\"your-upcloud-username\\"\\npassword = \\"your-upcloud-password\\" # Will be encrypted # Default settings\\ndefault_zone = \\"de-fra1\\"\\ndefault_plan = \\"2xCPU-4GB\\"\\n```plaintext **AWS Provider:** ```bash\\n# Create AWS config\\nvim workspace/config/providers/aws.toml\\n```plaintext ```toml\\n[aws]\\nregion = \\"us-east-1\\"\\naccess_key_id = \\"AKIAXXXXX\\"\\nsecret_access_key = \\"xxxxx\\" # Will be encrypted # Default settings\\ndefault_instance_type = \\"t3.medium\\"\\ndefault_region = \\"us-east-1\\"\\n```plaintext ### Encrypt Sensitive Data ```bash\\n# Generate Age key if not done already\\nage-keygen -o ~/.age/key.txt # Encrypt provider configs\\nkms encrypt (open workspace/config/providers/upcloud.toml) --backend age \\\\ | save workspace/config/providers/upcloud.toml.enc # Or use SOPS\\nsops --encrypt --age $(cat ~/.age/key.txt | grep \\"public key:\\" | cut -d: -f2) \\\\ workspace/config/providers/upcloud.toml > workspace/config/providers/upcloud.toml.enc # Remove plaintext\\nrm workspace/config/providers/upcloud.toml\\n```plaintext ### Configure Local Overrides ```bash\\n# Edit user-specific settings\\nvim workspace/config/local-overrides.toml\\n```plaintext ```toml\\n[user]\\nname = \\"admin\\"\\nemail = \\"admin@example.com\\" [preferences]\\neditor = \\"vim\\"\\noutput_format = \\"yaml\\"\\nconfirm_delete = true\\nconfirm_deploy = true [http]\\nuse_curl = true # Use curl instead of ureq [paths]\\nssh_key = \\"~/.ssh/id_ed25519\\"\\n```plaintext --- ## Step 7: Discover and Load Modules ### Discover Available Modules ```bash\\n# Discover task services\\nprovisioning module discover taskserv\\n# Shows: kubernetes, containerd, etcd, cilium, helm, etc. # Discover providers\\nprovisioning module discover provider\\n# Shows: upcloud, aws, local # Discover clusters\\nprovisioning module discover cluster\\n# Shows: buildkit, registry, monitoring, etc.\\n```plaintext ### Load Modules into Workspace ```bash\\n# Load Kubernetes taskserv\\nprovisioning module load taskserv production kubernetes # Load multiple modules\\nprovisioning module load taskserv production kubernetes containerd cilium # Load cluster configuration\\nprovisioning module load cluster production buildkit # Verify loaded modules\\nprovisioning module list taskserv production\\nprovisioning module list cluster production\\n```plaintext --- ## Step 8: Validate Configuration Before deploying, validate all configuration: ```bash\\n# Validate workspace configuration\\nprovisioning workspace validate # Validate infrastructure configuration\\nprovisioning validate config # Validate specific infrastructure\\nprovisioning infra validate --infra production # Check environment variables\\nprovisioning env # Show all configuration and environment\\nprovisioning allenv\\n```plaintext **Expected output:** ```plaintext\\n✓ Configuration valid\\n✓ Provider credentials configured\\n✓ Workspace initialized\\n✓ Modules loaded: 3 taskservs, 1 cluster\\n✓ SSH key configured\\n✓ Age encryption key available\\n```plaintext **Fix any errors** before proceeding to deployment. --- ## Step 9: Deploy Servers ### Preview Server Creation (Dry Run) ```bash\\n# Check what would be created (no actual changes)\\nprovisioning server create --infra production --check # With debug output for details\\nprovisioning server create --infra production --check --debug\\n```plaintext **Review the output:** - Server names and configurations\\n- Zones and regions\\n- CPU, memory, disk specifications\\n- Estimated costs\\n- Network settings ### Create Servers ```bash\\n# Create servers (with confirmation prompt)\\nprovisioning server create --infra production # Or auto-confirm (skip prompt)\\nprovisioning server create --infra production --yes # Wait for completion\\nprovisioning server create --infra production --wait\\n```plaintext **Expected output:** ```plaintext\\nCreating servers for infrastructure: production ● Creating server: k8s-master-01 (de-fra1, 4xCPU-8GB) ● Creating server: k8s-worker-01 (de-fra1, 4xCPU-8GB) ● Creating server: k8s-worker-02 (de-fra1, 4xCPU-8GB) ✓ Created 3 servers in 120 seconds Servers: • k8s-master-01: 192.168.1.10 (Running) • k8s-worker-01: 192.168.1.11 (Running) • k8s-worker-02: 192.168.1.12 (Running)\\n```plaintext ### Verify Server Creation ```bash\\n# List all servers\\nprovisioning server list --infra production # Show detailed server info\\nprovisioning server list --infra production --out yaml # SSH to server (test connectivity)\\nprovisioning server ssh k8s-master-01\\n# Type \'exit\' to return\\n```plaintext --- ## Step 10: Install Task Services Task services are infrastructure components like Kubernetes, databases, monitoring, etc. ### Install Kubernetes (Check Mode First) ```bash\\n# Preview Kubernetes installation\\nprovisioning taskserv create kubernetes --infra production --check # Shows:\\n# - Dependencies required (containerd, etcd)\\n# - Configuration to be applied\\n# - Resources needed\\n# - Estimated installation time\\n```plaintext ### Install Kubernetes ```bash\\n# Install Kubernetes (with dependencies)\\nprovisioning taskserv create kubernetes --infra production # Or install dependencies first\\nprovisioning taskserv create containerd --infra production\\nprovisioning taskserv create etcd --infra production\\nprovisioning taskserv create kubernetes --infra production # Monitor progress\\nprovisioning workflow monitor \\n```plaintext **Expected output:** ```plaintext\\nInstalling taskserv: kubernetes ● Installing containerd on k8s-master-01 ● Installing containerd on k8s-worker-01 ● Installing containerd on k8s-worker-02 ✓ Containerd installed (30s) ● Installing etcd on k8s-master-01 ✓ etcd installed (20s) ● Installing Kubernetes control plane on k8s-master-01 ✓ Kubernetes control plane ready (45s) ● Joining worker nodes ✓ k8s-worker-01 joined (15s) ✓ k8s-worker-02 joined (15s) ✓ Kubernetes installation complete (125 seconds) Cluster Info: • Version: 1.28.0 • Nodes: 3 (1 control-plane, 2 workers) • API Server: https://192.168.1.10:6443\\n```plaintext ### Install Additional Services ```bash\\n# Install Cilium (CNI)\\nprovisioning taskserv create cilium --infra production # Install Helm\\nprovisioning taskserv create helm --infra production # Verify all taskservs\\nprovisioning taskserv list --infra production\\n```plaintext --- ## Step 11: Create Clusters Clusters are complete application stacks (e.g., BuildKit, OCI Registry, Monitoring). ### Create BuildKit Cluster (Check Mode) ```bash\\n# Preview cluster creation\\nprovisioning cluster create buildkit --infra production --check # Shows:\\n# - Components to be deployed\\n# - Dependencies required\\n# - Configuration values\\n# - Resource requirements\\n```plaintext ### Create BuildKit Cluster ```bash\\n# Create BuildKit cluster\\nprovisioning cluster create buildkit --infra production # Monitor deployment\\nprovisioning workflow monitor  # Or use plugin for faster monitoring\\norch tasks --status running\\n```plaintext **Expected output:** ```plaintext\\nCreating cluster: buildkit ● Deploying BuildKit daemon ● Deploying BuildKit worker ● Configuring BuildKit cache ● Setting up BuildKit registry integration ✓ BuildKit cluster ready (60 seconds) Cluster Info: • BuildKit version: 0.12.0 • Workers: 2 • Cache: 50GB • Registry: registry.production.local\\n```plaintext ### Verify Cluster ```bash\\n# List all clusters\\nprovisioning cluster list --infra production # Show cluster details\\nprovisioning cluster list --infra production --out yaml # Check cluster health\\nkubectl get pods -n buildkit\\n```plaintext --- ## Step 12: Verify Deployment ### Comprehensive Health Check ```bash\\n# Check orchestrator status\\norch status\\n# or\\nprovisioning orchestrator status # Check all servers\\nprovisioning server list --infra production # Check all taskservs\\nprovisioning taskserv list --infra production # Check all clusters\\nprovisioning cluster list --infra production # Verify Kubernetes cluster\\nkubectl get nodes\\nkubectl get pods --all-namespaces\\n```plaintext ### Run Validation Tests ```bash\\n# Validate infrastructure\\nprovisioning infra validate --infra production # Test connectivity\\nprovisioning server ssh k8s-master-01 \\"kubectl get nodes\\" # Test BuildKit\\nkubectl exec -it -n buildkit buildkit-0 -- buildctl --version\\n```plaintext ### Expected Results All checks should show: - ✅ Servers: Running\\n- ✅ Taskservs: Installed and healthy\\n- ✅ Clusters: Deployed and operational\\n- ✅ Kubernetes: 3/3 nodes ready\\n- ✅ BuildKit: 2/2 workers ready --- ## Step 13: Post-Deployment ### Configure kubectl Access ```bash\\n# Get kubeconfig from master node\\nprovisioning server ssh k8s-master-01 \\"cat ~/.kube/config\\" > ~/.kube/config-production # Set KUBECONFIG\\nexport KUBECONFIG=~/.kube/config-production # Verify access\\nkubectl get nodes\\nkubectl get pods --all-namespaces\\n```plaintext ### Set Up Monitoring (Optional) ```bash\\n# Deploy monitoring stack\\nprovisioning cluster create monitoring --infra production # Access Grafana\\nkubectl port-forward -n monitoring svc/grafana 3000:80\\n# Open: http://localhost:3000\\n```plaintext ### Configure CI/CD Integration (Optional) ```bash\\n# Generate CI/CD credentials\\nprovisioning secrets generate aws --ttl 12h # Create CI/CD kubeconfig\\nkubectl create serviceaccount ci-cd -n default\\nkubectl create clusterrolebinding ci-cd --clusterrole=admin --serviceaccount=default:ci-cd\\n```plaintext ### Backup Configuration ```bash\\n# Backup workspace configuration\\ntar -czf workspace-production-backup.tar.gz workspace/ # Encrypt backup\\nkms encrypt (open workspace-production-backup.tar.gz | encode base64) --backend age \\\\ | save workspace-production-backup.tar.gz.enc # Store securely (S3, Vault, etc.)\\n```plaintext --- ## Troubleshooting ### Server Creation Fails **Problem**: Server creation times out or fails ```bash\\n# Check provider credentials\\nprovisioning validate config # Check provider API status\\ncurl -u username:password https://api.upcloud.com/1.3/account # Try with debug mode\\nprovisioning server create --infra production --check --debug\\n```plaintext ### Taskserv Installation Fails **Problem**: Kubernetes installation fails ```bash\\n# Check server connectivity\\nprovisioning server ssh k8s-master-01 # Check logs\\nprovisioning orchestrator logs | grep kubernetes # Check dependencies\\nprovisioning taskserv list --infra production | where status == \\"failed\\" # Retry installation\\nprovisioning taskserv delete kubernetes --infra production\\nprovisioning taskserv create kubernetes --infra production\\n```plaintext ### Plugin Commands Don\'t Work **Problem**: `auth`, `kms`, or `orch` commands not found ```bash\\n# Check plugin registration\\nplugin list | where name =~ \\"auth|kms|orch\\" # Re-register if missing\\ncd provisioning/core/plugins/nushell-plugins\\nplugin add target/release/nu_plugin_auth\\nplugin add target/release/nu_plugin_kms\\nplugin add target/release/nu_plugin_orchestrator # Restart Nushell\\nexit\\nnu\\n```plaintext ### KMS Encryption Fails **Problem**: `kms encrypt` returns error ```bash\\n# Check backend status\\nkms status # Check RustyVault running\\ncurl http://localhost:8200/v1/sys/health # Use Age backend instead (local)\\nkms encrypt \\"data\\" --backend age --key age1xxxxxxxxx # Check Age key\\ncat ~/.age/key.txt\\n```plaintext ### Orchestrator Not Running **Problem**: `orch status` returns error ```bash\\n# Check orchestrator status\\nps aux | grep orchestrator # Start orchestrator\\ncd provisioning/platform/orchestrator\\n./scripts/start-orchestrator.nu --background # Check logs\\ntail -f provisioning/platform/orchestrator/data/orchestrator.log\\n```plaintext ### Configuration Validation Errors **Problem**: `provisioning validate config` shows errors ```bash\\n# Show detailed errors\\nprovisioning validate config --debug # Check configuration files\\nprovisioning allenv # Fix missing settings\\nvim workspace/config/local-overrides.toml\\n```plaintext --- ## Next Steps ### Explore Advanced Features 1. **Multi-Environment Deployment** ```bash # Create dev and staging workspaces provisioning workspace create dev provisioning workspace create staging provisioning workspace switch dev Batch Operations # Deploy to multiple clouds\\nprovisioning batch submit workflows/multi-cloud-deploy.k Security Features # Enable MFA\\nauth mfa enroll totp # Set up break-glass\\nprovisioning break-glass request \\"Emergency access\\" Compliance and Audit # Generate compliance report\\nprovisioning compliance report --standard soc2","breadcrumbs":"From Scratch » macOS (via Homebrew)","id":"1735","title":"macOS (via Homebrew)"},"1736":{"body":"Quick Reference : provisioning sc or docs/guides/quickstart-cheatsheet.md Update Guide : docs/guides/update-infrastructure.md Customize Guide : docs/guides/customize-infrastructure.md Plugin Guide : docs/user/PLUGIN_INTEGRATION_GUIDE.md Security System : docs/architecture/ADR-009-security-system-complete.md","breadcrumbs":"From Scratch » Learn More","id":"1736","title":"Learn More"},"1737":{"body":"# Show help for any command\\nprovisioning help\\nprovisioning help server\\nprovisioning help taskserv # Check version\\nprovisioning version # Start Nushell session with provisioning library\\nprovisioning nu\\n```plaintext --- ## Summary You\'ve successfully: ✅ Installed Nushell and essential tools\\n✅ Built and registered native plugins (10-50x faster operations)\\n✅ Cloned and configured the project\\n✅ Initialized a production workspace\\n✅ Configured provider credentials\\n✅ Deployed servers\\n✅ Installed Kubernetes and task services\\n✅ Created application clusters\\n✅ Verified complete deployment **Your infrastructure is now ready for production use!** --- **Estimated Total Time**: 30-60 minutes\\n**Next Guide**: [Update Infrastructure](update-infrastructure.md)\\n**Questions?**: Open an issue or contact  **Last Updated**: 2025-10-09\\n**Version**: 3.5.0","breadcrumbs":"From Scratch » Get Help","id":"1737","title":"Get Help"},"1738":{"body":"Goal : Safely update running infrastructure with minimal downtime Time : 15-30 minutes Difficulty : Intermediate","breadcrumbs":"Update Infrastructure » Update Existing Infrastructure","id":"1738","title":"Update Existing Infrastructure"},"1739":{"body":"This guide covers: Checking for updates Planning update strategies Updating task services Rolling updates Rollback procedures Verification","breadcrumbs":"Update Infrastructure » Overview","id":"1739","title":"Overview"},"174":{"body":"","breadcrumbs":"First Deployment » Troubleshooting","id":"174","title":"Troubleshooting"},"1740":{"body":"","breadcrumbs":"Update Infrastructure » Update Strategies","id":"1740","title":"Update Strategies"},"1741":{"body":"Best for : Non-critical environments, development, staging # Direct update without downtime consideration\\nprovisioning t create  --infra \\n```plaintext ### Strategy 2: Rolling Updates (Recommended) **Best for**: Production environments, high availability ```bash\\n# Update servers one by one\\nprovisioning s update --infra  --rolling\\n```plaintext ### Strategy 3: Blue-Green Deployment (Safest) **Best for**: Critical production, zero-downtime requirements ```bash\\n# Create new infrastructure, switch traffic, remove old\\nprovisioning ws init -green\\n# ... configure and deploy\\n# ... switch traffic\\nprovisioning ws delete -blue\\n```plaintext ## Step 1: Check for Updates ### 1.1 Check All Task Services ```bash\\n# Check all taskservs for updates\\nprovisioning t check-updates\\n```plaintext **Expected Output:** ```plaintext\\n📦 Task Service Update Check: NAME CURRENT LATEST STATUS\\nkubernetes 1.29.0 1.30.0 ⬆️ update available\\ncontainerd 1.7.13 1.7.13 ✅ up-to-date\\ncilium 1.14.5 1.15.0 ⬆️ update available\\npostgres 15.5 16.1 ⬆️ update available\\nredis 7.2.3 7.2.3 ✅ up-to-date Updates available: 3\\n```plaintext ### 1.2 Check Specific Task Service ```bash\\n# Check specific taskserv\\nprovisioning t check-updates kubernetes\\n```plaintext **Expected Output:** ```plaintext\\n📦 Kubernetes Update Check: Current: 1.29.0\\nLatest: 1.30.0\\nStatus: ⬆️ Update available Changelog: • Enhanced security features • Performance improvements • Bug fixes in kube-apiserver • New workload resource types Breaking Changes: • None Recommended: ✅ Safe to update\\n```plaintext ### 1.3 Check Version Status ```bash\\n# Show detailed version information\\nprovisioning version show\\n```plaintext **Expected Output:** ```plaintext\\n📋 Component Versions: COMPONENT CURRENT LATEST DAYS OLD STATUS\\nkubernetes 1.29.0 1.30.0 45 ⬆️ update\\ncontainerd 1.7.13 1.7.13 0 ✅ current\\ncilium 1.14.5 1.15.0 30 ⬆️ update\\npostgres 15.5 16.1 60 ⬆️ update (major)\\nredis 7.2.3 7.2.3 0 ✅ current\\n```plaintext ### 1.4 Check for Security Updates ```bash\\n# Check for security-related updates\\nprovisioning version updates --security-only\\n```plaintext ## Step 2: Plan Your Update ### 2.1 Review Current Configuration ```bash\\n# Show current infrastructure\\nprovisioning show settings --infra my-production\\n```plaintext ### 2.2 Backup Configuration ```bash\\n# Create configuration backup\\ncp -r workspace/infra/my-production workspace/infra/my-production.backup-$(date +%Y%m%d) # Or use built-in backup\\nprovisioning ws backup my-production\\n```plaintext **Expected Output:** ```plaintext\\n✅ Backup created: workspace/backups/my-production-20250930.tar.gz\\n```plaintext ### 2.3 Create Update Plan ```bash\\n# Generate update plan\\nprovisioning plan update --infra my-production\\n```plaintext **Expected Output:** ```plaintext\\n📝 Update Plan for my-production: Phase 1: Minor Updates (Low Risk) • containerd: No update needed • redis: No update needed Phase 2: Patch Updates (Medium Risk) • cilium: 1.14.5 → 1.15.0 (estimated 5 minutes) Phase 3: Major Updates (High Risk - Requires Testing) • kubernetes: 1.29.0 → 1.30.0 (estimated 15 minutes) • postgres: 15.5 → 16.1 (estimated 10 minutes, may require data migration) Recommended Order: 1. Update cilium (low risk) 2. Update kubernetes (test in staging first) 3. Update postgres (requires maintenance window) Total Estimated Time: 30 minutes\\nRecommended: Test in staging environment first\\n```plaintext ## Step 3: Update Task Services ### 3.1 Update Non-Critical Service (Cilium Example) #### Dry-Run Update ```bash\\n# Test update without applying\\nprovisioning t create cilium --infra my-production --check\\n```plaintext **Expected Output:** ```plaintext\\n🔍 CHECK MODE: Simulating Cilium update Current: 1.14.5\\nTarget: 1.15.0 Would perform: 1. Download Cilium 1.15.0 2. Update configuration 3. Rolling restart of Cilium pods 4. Verify connectivity Estimated downtime: <1 minute per node\\nNo errors detected. Ready to update.\\n```plaintext #### Generate Updated Configuration ```bash\\n# Generate new configuration\\nprovisioning t generate cilium --infra my-production\\n```plaintext **Expected Output:** ```plaintext\\n✅ Generated Cilium configuration (version 1.15.0) Saved to: workspace/infra/my-production/taskservs/cilium.k\\n```plaintext #### Apply Update ```bash\\n# Apply update\\nprovisioning t create cilium --infra my-production\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Updating Cilium on my-production... Downloading Cilium 1.15.0... ⏳\\n✅ Downloaded Updating configuration... ⏳\\n✅ Configuration updated Rolling restart: web-01... ⏳\\n✅ web-01 updated (Cilium 1.15.0) Rolling restart: web-02... ⏳\\n✅ web-02 updated (Cilium 1.15.0) Verifying connectivity... ⏳\\n✅ All nodes connected 🎉 Cilium update complete! Version: 1.14.5 → 1.15.0 Downtime: 0 minutes\\n```plaintext #### Verify Update ```bash\\n# Verify updated version\\nprovisioning version taskserv cilium\\n```plaintext **Expected Output:** ```plaintext\\n📦 Cilium Version Info: Installed: 1.15.0\\nLatest: 1.15.0\\nStatus: ✅ Up-to-date Nodes: ✅ web-01: 1.15.0 (running) ✅ web-02: 1.15.0 (running)\\n```plaintext ### 3.2 Update Critical Service (Kubernetes Example) #### Test in Staging First ```bash\\n# If you have staging environment\\nprovisioning t create kubernetes --infra my-staging --check\\nprovisioning t create kubernetes --infra my-staging # Run integration tests\\nprovisioning test kubernetes --infra my-staging\\n```plaintext #### Backup Current State ```bash\\n# Backup Kubernetes state\\nkubectl get all -A -o yaml > k8s-backup-$(date +%Y%m%d).yaml # Backup etcd (if using external etcd)\\nprovisioning t backup kubernetes --infra my-production\\n```plaintext #### Schedule Maintenance Window ```bash\\n# Set maintenance mode (optional, if supported)\\nprovisioning maintenance enable --infra my-production --duration 30m\\n```plaintext #### Update Kubernetes ```bash\\n# Update control plane first\\nprovisioning t create kubernetes --infra my-production --control-plane-only\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Updating Kubernetes control plane on my-production... Draining control plane: web-01... ⏳\\n✅ web-01 drained Updating control plane: web-01... ⏳\\n✅ web-01 updated (Kubernetes 1.30.0) Uncordoning: web-01... ⏳\\n✅ web-01 ready Verifying control plane... ⏳\\n✅ Control plane healthy 🎉 Control plane update complete!\\n```plaintext ```bash\\n# Update worker nodes one by one\\nprovisioning t create kubernetes --infra my-production --workers-only --rolling\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Updating Kubernetes workers on my-production... Rolling update: web-02... Draining... ⏳ ✅ Drained (pods rescheduled) Updating... ⏳ ✅ Updated (Kubernetes 1.30.0) Uncordoning... ⏳ ✅ Ready Waiting for pods to stabilize... ⏳ ✅ All pods running 🎉 Worker update complete! Updated: web-02 Version: 1.30.0\\n```plaintext #### Verify Update ```bash\\n# Verify Kubernetes cluster\\nkubectl get nodes\\nprovisioning version taskserv kubernetes\\n```plaintext **Expected Output:** ```plaintext\\nNAME STATUS ROLES AGE VERSION\\nweb-01 Ready control-plane 30d v1.30.0\\nweb-02 Ready  30d v1.30.0\\n```plaintext ```bash\\n# Run smoke tests\\nprovisioning test kubernetes --infra my-production\\n```plaintext ### 3.3 Update Database (PostgreSQL Example) ⚠️ **WARNING**: Database updates may require data migration. Always backup first! #### Backup Database ```bash\\n# Backup PostgreSQL database\\nprovisioning t backup postgres --infra my-production\\n```plaintext **Expected Output:** ```plaintext\\n🗄️ Backing up PostgreSQL... Creating dump: my-production-postgres-20250930.sql... ⏳\\n✅ Dump created (2.3 GB) Compressing... ⏳\\n✅ Compressed (450 MB) Saved to: workspace/backups/postgres/my-production-20250930.sql.gz\\n```plaintext #### Check Compatibility ```bash\\n# Check if data migration is needed\\nprovisioning t check-migration postgres --from 15.5 --to 16.1\\n```plaintext **Expected Output:** ```plaintext\\n🔍 PostgreSQL Migration Check: From: 15.5\\nTo: 16.1 Migration Required: ✅ Yes (major version change) Steps Required: 1. Dump database with pg_dump 2. Stop PostgreSQL 15.5 3. Install PostgreSQL 16.1 4. Initialize new data directory 5. Restore from dump Estimated Time: 15-30 minutes (depending on data size)\\nEstimated Downtime: 15-30 minutes Recommended: Use streaming replication for zero-downtime upgrade\\n```plaintext #### Perform Update ```bash\\n# Update PostgreSQL (with automatic migration)\\nprovisioning t create postgres --infra my-production --migrate\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Updating PostgreSQL on my-production... ⚠️ Major version upgrade detected (15.5 → 16.1) Automatic migration will be performed Dumping database... ⏳\\n✅ Database dumped (2.3 GB) Stopping PostgreSQL 15.5... ⏳\\n✅ Stopped Installing PostgreSQL 16.1... ⏳\\n✅ Installed Initializing new data directory... ⏳\\n✅ Initialized Restoring database... ⏳\\n✅ Restored (2.3 GB) Starting PostgreSQL 16.1... ⏳\\n✅ Started Verifying data integrity... ⏳\\n✅ All tables verified 🎉 PostgreSQL update complete! Version: 15.5 → 16.1 Downtime: 18 minutes\\n```plaintext #### Verify Update ```bash\\n# Verify PostgreSQL\\nprovisioning version taskserv postgres\\nssh db-01 \\"psql --version\\"\\n```plaintext ## Step 4: Update Multiple Services ### 4.1 Batch Update (Sequentially) ```bash\\n# Update multiple taskservs one by one\\nprovisioning t update --infra my-production --taskservs cilium,containerd,redis\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Updating 3 taskservs on my-production... [1/3] Updating cilium... ⏳\\n✅ cilium updated (1.15.0) [2/3] Updating containerd... ⏳\\n✅ containerd updated (1.7.14) [3/3] Updating redis... ⏳\\n✅ redis updated (7.2.4) 🎉 All updates complete! Updated: 3 taskservs Total time: 8 minutes\\n```plaintext ### 4.2 Parallel Update (Non-Dependent Services) ```bash\\n# Update taskservs in parallel (if they don\'t depend on each other)\\nprovisioning t update --infra my-production --taskservs redis,postgres --parallel\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Updating 2 taskservs in parallel on my-production... redis: Updating... ⏳\\npostgres: Updating... ⏳ redis: ✅ Updated (7.2.4)\\npostgres: ✅ Updated (16.1) 🎉 All updates complete! Updated: 2 taskservs Total time: 3 minutes (parallel)\\n```plaintext ## Step 5: Update Server Configuration ### 5.1 Update Server Resources ```bash\\n# Edit server configuration\\nprovisioning sops workspace/infra/my-production/servers.k\\n```plaintext **Example: Upgrade server plan** ```kcl\\n# Before\\n{ name = \\"web-01\\" plan = \\"1xCPU-2GB\\" # Old plan\\n} # After\\n{ name = \\"web-01\\" plan = \\"2xCPU-4GB\\" # New plan\\n}\\n```plaintext ```bash\\n# Apply server update\\nprovisioning s update --infra my-production --check\\nprovisioning s update --infra my-production\\n```plaintext ### 5.2 Update Server OS ```bash\\n# Update operating system packages\\nprovisioning s update --infra my-production --os-update\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Updating OS packages on my-production servers... web-01: Updating packages... ⏳\\n✅ web-01: 24 packages updated web-02: Updating packages... ⏳\\n✅ web-02: 24 packages updated db-01: Updating packages... ⏳\\n✅ db-01: 24 packages updated 🎉 OS updates complete!\\n```plaintext ## Step 6: Rollback Procedures ### 6.1 Rollback Task Service If update fails or causes issues: ```bash\\n# Rollback to previous version\\nprovisioning t rollback cilium --infra my-production\\n```plaintext **Expected Output:** ```plaintext\\n🔄 Rolling back Cilium on my-production... Current: 1.15.0\\nTarget: 1.14.5 (previous version) Rolling back: web-01... ⏳\\n✅ web-01 rolled back Rolling back: web-02... ⏳\\n✅ web-02 rolled back Verifying connectivity... ⏳\\n✅ All nodes connected 🎉 Rollback complete! Version: 1.15.0 → 1.14.5\\n```plaintext ### 6.2 Rollback from Backup ```bash\\n# Restore configuration from backup\\nprovisioning ws restore my-production --from workspace/backups/my-production-20250930.tar.gz\\n```plaintext ### 6.3 Emergency Rollback ```bash\\n# Complete infrastructure rollback\\nprovisioning rollback --infra my-production --to-snapshot \\n```plaintext ## Step 7: Post-Update Verification ### 7.1 Verify All Components ```bash\\n# Check overall health\\nprovisioning health --infra my-production\\n```plaintext **Expected Output:** ```plaintext\\n🏥 Health Check: my-production Servers: ✅ web-01: Healthy ✅ web-02: Healthy ✅ db-01: Healthy Task Services: ✅ kubernetes: 1.30.0 (healthy) ✅ containerd: 1.7.13 (healthy) ✅ cilium: 1.15.0 (healthy) ✅ postgres: 16.1 (healthy) Clusters: ✅ buildkit: 2/2 replicas (healthy) Overall Status: ✅ All systems healthy\\n```plaintext ### 7.2 Verify Version Updates ```bash\\n# Verify all versions are updated\\nprovisioning version show\\n```plaintext ### 7.3 Run Integration Tests ```bash\\n# Run comprehensive tests\\nprovisioning test all --infra my-production\\n```plaintext **Expected Output:** ```plaintext\\n🧪 Running Integration Tests... [1/5] Server connectivity... ⏳\\n✅ All servers reachable [2/5] Kubernetes health... ⏳\\n✅ All nodes ready, all pods running [3/5] Network connectivity... ⏳\\n✅ All services reachable [4/5] Database connectivity... ⏳\\n✅ PostgreSQL responsive [5/5] Application health... ⏳\\n✅ All applications healthy 🎉 All tests passed!\\n```plaintext ### 7.4 Monitor for Issues ```bash\\n# Monitor logs for errors\\nprovisioning logs --infra my-production --follow --level error\\n```plaintext ## Update Checklist Use this checklist for production updates: - [ ] Check for available updates\\n- [ ] Review changelog and breaking changes\\n- [ ] Create configuration backup\\n- [ ] Test update in staging environment\\n- [ ] Schedule maintenance window\\n- [ ] Notify team/users of maintenance\\n- [ ] Update non-critical services first\\n- [ ] Verify each update before proceeding\\n- [ ] Update critical services with rolling updates\\n- [ ] Backup database before major updates\\n- [ ] Verify all components after update\\n- [ ] Run integration tests\\n- [ ] Monitor for issues (30 minutes minimum)\\n- [ ] Document any issues encountered\\n- [ ] Close maintenance window ## Common Update Scenarios ### Scenario 1: Minor Security Patch ```bash\\n# Quick security update\\nprovisioning t check-updates --security-only\\nprovisioning t update --infra my-production --security-patches --yes\\n```plaintext ### Scenario 2: Major Version Upgrade ```bash\\n# Careful major version update\\nprovisioning ws backup my-production\\nprovisioning t check-migration  --from X.Y --to X+1.Y\\nprovisioning t create  --infra my-production --migrate\\nprovisioning test all --infra my-production\\n```plaintext ### Scenario 3: Emergency Hotfix ```bash\\n# Apply critical hotfix immediately\\nprovisioning t create  --infra my-production --hotfix --yes\\n```plaintext ## Troubleshooting Updates ### Issue: Update fails mid-process **Solution:** ```bash\\n# Check update status\\nprovisioning t status  --infra my-production # Resume failed update\\nprovisioning t update  --infra my-production --resume # Or rollback\\nprovisioning t rollback  --infra my-production\\n```plaintext ### Issue: Service not starting after update **Solution:** ```bash\\n# Check logs\\nprovisioning logs  --infra my-production # Verify configuration\\nprovisioning t validate  --infra my-production # Rollback if necessary\\nprovisioning t rollback  --infra my-production\\n```plaintext ### Issue: Data migration fails **Solution:** ```bash\\n# Check migration logs\\nprovisioning t migration-logs  --infra my-production # Restore from backup\\nprovisioning t restore  --infra my-production --from \\n```plaintext ## Best Practices 1. **Always Test First**: Test updates in staging before production\\n2. **Backup Everything**: Create backups before any update\\n3. **Update Gradually**: Update one service at a time\\n4. **Monitor Closely**: Watch for errors after each update\\n5. **Have Rollback Plan**: Always have a rollback strategy\\n6. **Document Changes**: Keep update logs for reference\\n7. **Schedule Wisely**: Update during low-traffic periods\\n8. **Verify Thoroughly**: Run tests after each update ## Next Steps - **[Customize Guide](customize-infrastructure.md)** - Customize your infrastructure\\n- **[From Scratch Guide](from-scratch.md)** - Deploy new infrastructure\\n- **[Workflow Guide](../development/workflow.md)** - Automate with workflows ## Quick Reference ```bash\\n# Update workflow\\nprovisioning t check-updates\\nprovisioning ws backup my-production\\nprovisioning t create  --infra my-production --check\\nprovisioning t create  --infra my-production\\nprovisioning version taskserv \\nprovisioning health --infra my-production\\nprovisioning test all --infra my-production\\n```plaintext --- *This guide is part of the provisioning project documentation. Last updated: 2025-09-30*","breadcrumbs":"Update Infrastructure » Strategy 1: In-Place Updates (Fastest)","id":"1741","title":"Strategy 1: In-Place Updates (Fastest)"},"1742":{"body":"Goal : Customize infrastructure using layers, templates, and configuration patterns Time : 20-40 minutes Difficulty : Intermediate to Advanced","breadcrumbs":"Customize Infrastructure » Customize Infrastructure","id":"1742","title":"Customize Infrastructure"},"1743":{"body":"This guide covers: Understanding the layer system Using templates Creating custom modules Configuration inheritance Advanced customization patterns","breadcrumbs":"Customize Infrastructure » Overview","id":"1743","title":"Overview"},"1744":{"body":"","breadcrumbs":"Customize Infrastructure » The Layer System","id":"1744","title":"The Layer System"},"1745":{"body":"The provisioning system uses a 3-layer architecture for configuration inheritance: ┌─────────────────────────────────────┐\\n│ Infrastructure Layer (Priority 300)│ ← Highest priority\\n│ workspace/infra/{name}/ │\\n│ • Project-specific configs │\\n│ • Environment customizations │\\n│ • Local overrides │\\n└─────────────────────────────────────┘ ↓ overrides\\n┌─────────────────────────────────────┐\\n│ Workspace Layer (Priority 200) │\\n│ provisioning/workspace/templates/ │\\n│ • Reusable patterns │\\n│ • Organization standards │\\n│ • Team conventions │\\n└─────────────────────────────────────┘ ↓ overrides\\n┌─────────────────────────────────────┐\\n│ Core Layer (Priority 100) │ ← Lowest priority\\n│ provisioning/extensions/ │\\n│ • System defaults │\\n│ • Provider implementations │\\n│ • Default taskserv configs │\\n└─────────────────────────────────────┘\\n```plaintext **Resolution Order**: Infrastructure (300) → Workspace (200) → Core (100) Higher numbers override lower numbers. ### View Layer Resolution ```bash\\n# Explain layer concept\\nprovisioning lyr explain\\n```plaintext **Expected Output:** ```plaintext\\n📚 LAYER SYSTEM EXPLAINED The layer system provides configuration inheritance across 3 levels: 🔵 CORE LAYER (100) - System Defaults Location: provisioning/extensions/ • Base taskserv configurations • Default provider settings • Standard cluster templates • Built-in extensions 🟢 WORKSPACE LAYER (200) - Shared Templates Location: provisioning/workspace/templates/ • Organization-wide patterns • Reusable configurations • Team standards • Custom extensions 🔴 INFRASTRUCTURE LAYER (300) - Project Specific Location: workspace/infra/{project}/ • Project-specific overrides • Environment customizations • Local modifications • Runtime settings Resolution: Infrastructure → Workspace → Core\\nHigher priority layers override lower ones.\\n```plaintext ```bash\\n# Show layer resolution for your project\\nprovisioning lyr show my-production\\n```plaintext **Expected Output:** ```plaintext\\n📊 Layer Resolution for my-production: LAYER PRIORITY SOURCE FILES\\nInfrastructure 300 workspace/infra/my-production/ 4 files • servers.k (overrides) • taskservs.k (overrides) • clusters.k (custom) • providers.k (overrides) Workspace 200 provisioning/workspace/templates/ 2 files • production.k (used) • kubernetes.k (used) Core 100 provisioning/extensions/ 15 files • taskservs/* (base configs) • providers/* (default settings) • clusters/* (templates) Resolution Order: Infrastructure → Workspace → Core\\nStatus: ✅ All layers resolved successfully\\n```plaintext ### Test Layer Resolution ```bash\\n# Test how a specific module resolves\\nprovisioning lyr test kubernetes my-production\\n```plaintext **Expected Output:** ```plaintext\\n🔍 Layer Resolution Test: kubernetes → my-production Resolving kubernetes configuration... 🔴 Infrastructure Layer (300): ✅ Found: workspace/infra/my-production/taskservs/kubernetes.k Provides: • version = \\"1.30.0\\" (overrides) • control_plane_servers = [\\"web-01\\"] (overrides) • worker_servers = [\\"web-02\\"] (overrides) 🟢 Workspace Layer (200): ✅ Found: provisioning/workspace/templates/production-kubernetes.k Provides: • security_policies (inherited) • network_policies (inherited) • resource_quotas (inherited) 🔵 Core Layer (100): ✅ Found: provisioning/extensions/taskservs/kubernetes/config.k Provides: • default_version = \\"1.29.0\\" (base) • default_features (base) • default_plugins (base) Final Configuration (after merging all layers): version: \\"1.30.0\\" (from Infrastructure) control_plane_servers: [\\"web-01\\"] (from Infrastructure) worker_servers: [\\"web-02\\"] (from Infrastructure) security_policies: {...} (from Workspace) network_policies: {...} (from Workspace) resource_quotas: {...} (from Workspace) default_features: {...} (from Core) default_plugins: {...} (from Core) Resolution: ✅ Success\\n```plaintext ## Using Templates ### List Available Templates ```bash\\n# List all templates\\nprovisioning tpl list\\n```plaintext **Expected Output:** ```plaintext\\n📋 Available Templates: TASKSERVS: • production-kubernetes - Production-ready Kubernetes setup • production-postgres - Production PostgreSQL with replication • production-redis - Redis cluster with sentinel • development-kubernetes - Development Kubernetes (minimal) • ci-cd-pipeline - Complete CI/CD pipeline PROVIDERS: • upcloud-production - UpCloud production settings • upcloud-development - UpCloud development settings • aws-production - AWS production VPC setup • aws-development - AWS development environment • local-docker - Local Docker-based setup CLUSTERS: • buildkit-cluster - BuildKit for container builds • monitoring-stack - Prometheus + Grafana + Loki • security-stack - Security monitoring tools Total: 13 templates\\n```plaintext ```bash\\n# List templates by type\\nprovisioning tpl list --type taskservs\\nprovisioning tpl list --type providers\\nprovisioning tpl list --type clusters\\n```plaintext ### View Template Details ```bash\\n# Show template details\\nprovisioning tpl show production-kubernetes\\n```plaintext **Expected Output:** ```plaintext\\n📄 Template: production-kubernetes Description: Production-ready Kubernetes configuration with security hardening, network policies, and monitoring Category: taskservs\\nVersion: 1.0.0 Configuration Provided: • Kubernetes version: 1.30.0 • Security policies: Pod Security Standards (restricted) • Network policies: Default deny + allow rules • Resource quotas: Per-namespace limits • Monitoring: Prometheus integration • Logging: Loki integration • Backup: Velero configuration Requirements: • Minimum 2 servers • 4GB RAM per server • Network plugin (Cilium recommended) Location: provisioning/workspace/templates/production-kubernetes.k Example Usage: provisioning tpl apply production-kubernetes my-production\\n```plaintext ### Apply Template ```bash\\n# Apply template to your infrastructure\\nprovisioning tpl apply production-kubernetes my-production\\n```plaintext **Expected Output:** ```plaintext\\n🚀 Applying template: production-kubernetes → my-production Checking compatibility... ⏳\\n✅ Infrastructure compatible with template Merging configuration... ⏳\\n✅ Configuration merged Files created/updated: • workspace/infra/my-production/taskservs/kubernetes.k (updated) • workspace/infra/my-production/policies/security.k (created) • workspace/infra/my-production/policies/network.k (created) • workspace/infra/my-production/monitoring/prometheus.k (created) 🎉 Template applied successfully! Next steps: 1. Review generated configuration 2. Adjust as needed 3. Deploy: provisioning t create kubernetes --infra my-production\\n```plaintext ### Validate Template Usage ```bash\\n# Validate template was applied correctly\\nprovisioning tpl validate my-production\\n```plaintext **Expected Output:** ```plaintext\\n✅ Template Validation: my-production Templates Applied: ✅ production-kubernetes (v1.0.0) ✅ production-postgres (v1.0.0) Configuration Status: ✅ All required fields present ✅ No conflicting settings ✅ Dependencies satisfied Compliance: ✅ Security policies configured ✅ Network policies configured ✅ Resource quotas set ✅ Monitoring enabled Status: ✅ Valid\\n```plaintext ## Creating Custom Templates ### Step 1: Create Template Structure ```bash\\n# Create custom template directory\\nmkdir -p provisioning/workspace/templates/my-custom-template\\n```plaintext ### Step 2: Write Template Configuration **File: `provisioning/workspace/templates/my-custom-template/config.k`** ```kcl\\n# Custom Kubernetes template with specific settings kubernetes_config = { # Version version = \\"1.30.0\\" # Custom feature gates feature_gates = { \\"GracefulNodeShutdown\\" = True \\"SeccompDefault\\" = True \\"StatefulSetAutoDeletePVC\\" = True } # Custom kubelet configuration kubelet_config = { max_pods = 110 pod_pids_limit = 4096 container_log_max_size = \\"10Mi\\" container_log_max_files = 5 } # Custom API server flags apiserver_extra_args = { \\"enable-admission-plugins\\" = \\"NodeRestriction,PodSecurity,LimitRanger\\" \\"audit-log-maxage\\" = \\"30\\" \\"audit-log-maxbackup\\" = \\"10\\" } # Custom scheduler configuration scheduler_config = { profiles = [ { name = \\"high-availability\\" plugins = { score = { enabled = [ {name = \\"NodeResourcesBalancedAllocation\\", weight = 2} {name = \\"NodeResourcesLeastAllocated\\", weight = 1} ] } } } ] } # Network configuration network = { service_cidr = \\"10.96.0.0/12\\" pod_cidr = \\"10.244.0.0/16\\" dns_domain = \\"cluster.local\\" } # Security configuration security = { pod_security_standard = \\"restricted\\" encrypt_etcd = True rotate_certificates = True }\\n}\\n```plaintext ### Step 3: Create Template Metadata **File: `provisioning/workspace/templates/my-custom-template/metadata.toml`** ```toml\\n[template]\\nname = \\"my-custom-template\\"\\nversion = \\"1.0.0\\"\\ndescription = \\"Custom Kubernetes template with enhanced security\\"\\ncategory = \\"taskservs\\"\\nauthor = \\"Your Name\\" [requirements]\\nmin_servers = 2\\nmin_memory_gb = 4\\nrequired_taskservs = [\\"containerd\\", \\"cilium\\"] [tags]\\nenvironment = [\\"production\\", \\"staging\\"]\\nfeatures = [\\"security\\", \\"monitoring\\", \\"high-availability\\"]\\n```plaintext ### Step 4: Test Custom Template ```bash\\n# List templates (should include your custom template)\\nprovisioning tpl list # Show your template\\nprovisioning tpl show my-custom-template # Apply to test infrastructure\\nprovisioning tpl apply my-custom-template my-test\\n```plaintext ## Configuration Inheritance Examples ### Example 1: Override Single Value **Core Layer** (`provisioning/extensions/taskservs/postgres/config.k`): ```kcl\\npostgres_config = { version = \\"15.5\\" port = 5432 max_connections = 100\\n}\\n```plaintext **Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.k`): ```kcl\\npostgres_config = { max_connections = 500 # Override only max_connections\\n}\\n```plaintext **Result** (after layer resolution): ```kcl\\npostgres_config = { version = \\"15.5\\" # From Core port = 5432 # From Core max_connections = 500 # From Infrastructure (overridden)\\n}\\n```plaintext ### Example 2: Add Custom Configuration **Workspace Layer** (`provisioning/workspace/templates/production-postgres.k`): ```kcl\\npostgres_config = { replication = { enabled = True replicas = 2 sync_mode = \\"async\\" }\\n}\\n```plaintext **Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.k`): ```kcl\\npostgres_config = { replication = { sync_mode = \\"sync\\" # Override sync mode } custom_extensions = [\\"pgvector\\", \\"timescaledb\\"] # Add custom config\\n}\\n```plaintext **Result**: ```kcl\\npostgres_config = { version = \\"15.5\\" # From Core port = 5432 # From Core max_connections = 100 # From Core replication = { enabled = True # From Workspace replicas = 2 # From Workspace sync_mode = \\"sync\\" # From Infrastructure (overridden) } custom_extensions = [\\"pgvector\\", \\"timescaledb\\"] # From Infrastructure (added)\\n}\\n```plaintext ### Example 3: Environment-Specific Configuration **Workspace Layer** (`provisioning/workspace/templates/base-kubernetes.k`): ```kcl\\nkubernetes_config = { version = \\"1.30.0\\" control_plane_count = 3 worker_count = 5 resources = { control_plane = {cpu = \\"4\\", memory = \\"8Gi\\"} worker = {cpu = \\"8\\", memory = \\"16Gi\\"} }\\n}\\n```plaintext **Development Infrastructure** (`workspace/infra/my-dev/taskservs/kubernetes.k`): ```kcl\\nkubernetes_config = { control_plane_count = 1 # Smaller for dev worker_count = 2 resources = { control_plane = {cpu = \\"2\\", memory = \\"4Gi\\"} worker = {cpu = \\"2\\", memory = \\"4Gi\\"} }\\n}\\n```plaintext **Production Infrastructure** (`workspace/infra/my-prod/taskservs/kubernetes.k`): ```kcl\\nkubernetes_config = { control_plane_count = 5 # Larger for prod worker_count = 10 resources = { control_plane = {cpu = \\"8\\", memory = \\"16Gi\\"} worker = {cpu = \\"16\\", memory = \\"32Gi\\"} }\\n}\\n```plaintext ## Advanced Customization Patterns ### Pattern 1: Multi-Environment Setup Create different configurations for each environment: ```bash\\n# Create environments\\nprovisioning ws init my-app-dev\\nprovisioning ws init my-app-staging\\nprovisioning ws init my-app-prod # Apply environment-specific templates\\nprovisioning tpl apply development-kubernetes my-app-dev\\nprovisioning tpl apply staging-kubernetes my-app-staging\\nprovisioning tpl apply production-kubernetes my-app-prod # Customize each environment\\n# Edit: workspace/infra/my-app-dev/...\\n# Edit: workspace/infra/my-app-staging/...\\n# Edit: workspace/infra/my-app-prod/...\\n```plaintext ### Pattern 2: Shared Configuration Library Create reusable configuration fragments: **File: `provisioning/workspace/templates/shared/security-policies.k`** ```kcl\\nsecurity_policies = { pod_security = { enforce = \\"restricted\\" audit = \\"restricted\\" warn = \\"restricted\\" } network_policies = [ { name = \\"deny-all\\" pod_selector = {} policy_types = [\\"Ingress\\", \\"Egress\\"] }, { name = \\"allow-dns\\" pod_selector = {} egress = [ { to = [{namespace_selector = {name = \\"kube-system\\"}}] ports = [{protocol = \\"UDP\\", port = 53}] } ] } ]\\n}\\n```plaintext Import in your infrastructure: ```kcl\\nimport \\"../../../provisioning/workspace/templates/shared/security-policies.k\\" kubernetes_config = { version = \\"1.30.0\\" # ... other config security = security_policies # Import shared policies\\n}\\n```plaintext ### Pattern 3: Dynamic Configuration Use KCL features for dynamic configuration: ```kcl\\n# Calculate resources based on server count\\nserver_count = 5\\nreplicas_per_server = 2\\ntotal_replicas = server_count * replicas_per_server postgres_config = { version = \\"16.1\\" max_connections = total_replicas * 50 # Dynamic calculation shared_buffers = \\"${total_replicas * 128}MB\\"\\n}\\n```plaintext ### Pattern 4: Conditional Configuration ```kcl\\nenvironment = \\"production\\" # or \\"development\\" kubernetes_config = { version = \\"1.30.0\\" control_plane_count = if environment == \\"production\\" { 3 } else { 1 } worker_count = if environment == \\"production\\" { 5 } else { 2 } monitoring = { enabled = environment == \\"production\\" retention = if environment == \\"production\\" { \\"30d\\" } else { \\"7d\\" } }\\n}\\n```plaintext ## Layer Statistics ```bash\\n# Show layer system statistics\\nprovisioning lyr stats\\n```plaintext **Expected Output:** ```plaintext\\n📊 Layer System Statistics: Infrastructure Layer: • Projects: 3 • Total files: 15 • Average overrides per project: 5 Workspace Layer: • Templates: 13 • Most used: production-kubernetes (5 projects) • Custom templates: 2 Core Layer: • Taskservs: 15 • Providers: 3 • Clusters: 3 Resolution Performance: • Average resolution time: 45ms • Cache hit rate: 87% • Total resolutions: 1,250\\n```plaintext ## Customization Workflow ### Complete Customization Example ```bash\\n# 1. Create new infrastructure\\nprovisioning ws init my-custom-app # 2. Understand layer system\\nprovisioning lyr explain # 3. Discover templates\\nprovisioning tpl list --type taskservs # 4. Apply base template\\nprovisioning tpl apply production-kubernetes my-custom-app # 5. View applied configuration\\nprovisioning lyr show my-custom-app # 6. Customize (edit files)\\nprovisioning sops workspace/infra/my-custom-app/taskservs/kubernetes.k # 7. Test layer resolution\\nprovisioning lyr test kubernetes my-custom-app # 8. Validate configuration\\nprovisioning tpl validate my-custom-app\\nprovisioning val config --infra my-custom-app # 9. Deploy customized infrastructure\\nprovisioning s create --infra my-custom-app --check\\nprovisioning s create --infra my-custom-app\\nprovisioning t create kubernetes --infra my-custom-app\\n```plaintext ## Best Practices ### 1. Use Layers Correctly - **Core Layer**: Only modify for system-wide changes\\n- **Workspace Layer**: Use for organization-wide templates\\n- **Infrastructure Layer**: Use for project-specific customizations ### 2. Template Organization ```plaintext\\nprovisioning/workspace/templates/\\n├── shared/ # Shared configuration fragments\\n│ ├── security-policies.k\\n│ ├── network-policies.k\\n│ └── monitoring.k\\n├── production/ # Production templates\\n│ ├── kubernetes.k\\n│ ├── postgres.k\\n│ └── redis.k\\n└── development/ # Development templates ├── kubernetes.k └── postgres.k\\n```plaintext ### 3. Documentation Document your customizations: **File: `workspace/infra/my-production/README.md`** ```markdown\\n# My Production Infrastructure ## Customizations - Kubernetes: Using production template with 5 control plane nodes\\n- PostgreSQL: Configured with streaming replication\\n- Cilium: Native routing mode enabled ## Layer Overrides - `taskservs/kubernetes.k`: Control plane count (3 → 5)\\n- `taskservs/postgres.k`: Replication mode (async → sync)\\n- `network/cilium.k`: Routing mode (tunnel → native)\\n```plaintext ### 4. Version Control Keep templates and configurations in version control: ```bash\\ncd provisioning/workspace/templates/\\ngit add .\\ngit commit -m \\"Add production Kubernetes template with enhanced security\\" cd workspace/infra/my-production/\\ngit add .\\ngit commit -m \\"Configure production environment for my-production\\"\\n```plaintext ## Troubleshooting Customizations ### Issue: Configuration not applied ```bash\\n# Check layer resolution\\nprovisioning lyr show my-production # Verify file exists\\nls -la workspace/infra/my-production/taskservs/ # Test specific resolution\\nprovisioning lyr test kubernetes my-production\\n```plaintext ### Issue: Conflicting configurations ```bash\\n# Validate configuration\\nprovisioning val config --infra my-production # Show configuration merge result\\nprovisioning show config kubernetes --infra my-production\\n```plaintext ### Issue: Template not found ```bash\\n# List available templates\\nprovisioning tpl list # Check template path\\nls -la provisioning/workspace/templates/ # Refresh template cache\\nprovisioning tpl refresh\\n```plaintext ## Next Steps - **[From Scratch Guide](from-scratch.md)** - Deploy new infrastructure\\n- **[Update Guide](update-infrastructure.md)** - Update existing infrastructure\\n- **[Workflow Guide](../development/workflow.md)** - Automate with workflows\\n- **[KCL Guide](../development/KCL_MODULE_GUIDE.md)** - Learn KCL configuration language ## Quick Reference ```bash\\n# Layer system\\nprovisioning lyr explain # Explain layers\\nprovisioning lyr show  # Show layer resolution\\nprovisioning lyr test   # Test resolution\\nprovisioning lyr stats # Layer statistics # Templates\\nprovisioning tpl list # List all templates\\nprovisioning tpl list --type  # Filter by type\\nprovisioning tpl show
Workflow Dashboard

{task.name}
