# Complete Deployment Guide: From Scratch to Production **Version**: 3.5.0 **Last Updated**: 2025-10-09 **Estimated Time**: 30-60 minutes **Difficulty**: Beginner to Intermediate --- ## Table of Contents 1. [Prerequisites](#prerequisites) 2. [Step 1: Install Nushell](#step-1-install-nushell) 3. [Step 2: Install Nushell Plugins (Recommended)](#step-2-install-nushell-plugins-recommended) 4. [Step 3: Install Required Tools](#step-3-install-required-tools) 5. [Step 4: Clone and Setup Project](#step-4-clone-and-setup-project) 6. [Step 5: Initialize Workspace](#step-5-initialize-workspace) 7. [Step 6: Configure Environment](#step-6-configure-environment) 8. [Step 7: Discover and Load Modules](#step-7-discover-and-load-modules) 9. [Step 8: Validate Configuration](#step-8-validate-configuration) 10. [Step 9: Deploy Servers](#step-9-deploy-servers) 11. [Step 10: Install Task Services](#step-10-install-task-services) 12. [Step 11: Create Clusters](#step-11-create-clusters) 13. [Step 12: Verify Deployment](#step-12-verify-deployment) 14. [Step 13: Post-Deployment](#step-13-post-deployment) 15. [Troubleshooting](#troubleshooting) 16. [Next Steps](#next-steps) --- ## Prerequisites Before starting, ensure you have: - ✅ **Operating System**: macOS, Linux, or Windows (WSL2 recommended) - ✅ **Administrator Access**: Ability to install software and configure system - ✅ **Internet Connection**: For downloading dependencies and accessing cloud providers - ✅ **Cloud Provider Credentials**: UpCloud, Hetzner, AWS, or local development environment - ✅ **Basic Terminal Knowledge**: Comfortable running shell commands - ✅ **Text Editor**: vim, nano, Zed, VSCode, or your preferred editor ### Recommended Hardware - **CPU**: 2+ cores - **RAM**: 8 GB minimum, 16 GB recommended - **Disk**: 20 GB free space minimum --- ## Step 1: Install Nushell Nushell 0.109.1+ is the primary shell and scripting language for the provisioning platform. ### macOS (via Homebrew) ```bash # Install Nushell brew install nushell # Verify installation nu --version # Expected: 0.109.1 or higher ``` ### Linux (via Package Manager) **Ubuntu/Debian:** ```bash # Add Nushell repository curl -fsSL https://starship.rs/install.sh | bash # Install Nushell sudo apt update sudo apt install nushell # Verify installation nu --version ``` **Fedora:** ```bash sudo dnf install nushell nu --version ``` **Arch Linux:** ```bash sudo pacman -S nushell nu --version ``` ### Linux/macOS (via Cargo) ```bash # Install Rust (if not already installed) curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env # Install Nushell cargo install nu --locked # Verify installation nu --version ``` ### Windows (via Winget) ```powershell # Install Nushell winget install nushell # Verify installation nu --version ``` ### Configure Nushell ```bash # Start Nushell nu # Configure (creates default config if not exists) config nu ``` --- ## Step 2: Install Nushell Plugins (Recommended) Native plugins provide **10-50x performance improvement** for authentication, KMS, and orchestrator operations. ### Why Install Plugins **Performance Gains:** - 🚀 **KMS operations**: ~5 ms vs ~50 ms (10x faster) - 🚀 **Orchestrator queries**: ~1 ms vs ~30 ms (30x faster) - 🚀 **Batch encryption**: 100 files in 0.5s vs 5s (10x faster) **Benefits:** - ✅ Native Nushell integration (pipelines, data structures) - ✅ OS keyring for secure token storage - ✅ Offline capability (Age encryption, local orchestrator) - ✅ Graceful fallback to HTTP if not installed ### Prerequisites for Building Plugins ```bash # Install Rust toolchain (if not already installed) curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env rustc --version # Expected: rustc 1.75+ or higher # Linux only: Install development packages sudo apt install libssl-dev pkg-config # Ubuntu/Debian sudo dnf install openssl-devel # Fedora # Linux only: Install keyring service (required for auth plugin) sudo apt install gnome-keyring # Ubuntu/Debian (GNOME) sudo apt install kwalletmanager # Ubuntu/Debian (KDE) ``` ### Build Plugins ```bash # Navigate to plugins directory cd provisioning/core/plugins/nushell-plugins # Build all three plugins in release mode (optimized) cargo build --release --all # Expected output: # Compiling nu_plugin_auth v0.1.0 # Compiling nu_plugin_kms v0.1.0 # Compiling nu_plugin_orchestrator v0.1.0 # Finished release [optimized] target(s) in 2m 15s ``` **Build time**: ~2-5 minutes depending on hardware ### Register Plugins with Nushell ```bash # Register all three plugins (full paths recommended) plugin add $PWD/target/release/nu_plugin_auth plugin add $PWD/target/release/nu_plugin_kms plugin add $PWD/target/release/nu_plugin_orchestrator # Alternative (from plugins directory) plugin add target/release/nu_plugin_auth plugin add target/release/nu_plugin_kms plugin add target/release/nu_plugin_orchestrator ``` ### Verify Plugin Installation ```bash # List registered plugins plugin list | where name =~ "auth|kms|orch" # Expected output: # ╭───┬─────────────────────────┬─────────┬───────────────────────────────────╮ # │ # │ name │ version │ filename │ # ├───┼─────────────────────────┼─────────┼───────────────────────────────────┤ # │ 0 │ nu_plugin_auth │ 0.1.0 │ .../nu_plugin_auth │ # │ 1 │ nu_plugin_kms │ 0.1.0 │ .../nu_plugin_kms │ # │ 2 │ nu_plugin_orchestrator │ 0.1.0 │ .../nu_plugin_orchestrator │ # ╰───┴─────────────────────────┴─────────┴───────────────────────────────────╯ # Test each plugin auth --help # Should show auth commands kms --help # Should show kms commands orch --help # Should show orch commands ``` ### Configure Plugin Environments ```bash # Add to ~/.config/nushell/env.nu $env.CONTROL_CENTER_URL = "http://localhost:3000" $env.RUSTYVAULT_ADDR = "http://localhost:8200" $env.RUSTYVAULT_TOKEN = "your-vault-token-here" $env.ORCHESTRATOR_DATA_DIR = "provisioning/platform/orchestrator/data" # For Age encryption (local development) $env.AGE_IDENTITY = $"($env.HOME)/.age/key.txt" $env.AGE_RECIPIENT = "age1xxxxxxxxx" # Replace with your public key ``` ### Test Plugins (Quick Smoke Test) ```bash # Test KMS plugin (requires backend configured) kms status # Expected: { backend: "rustyvault", status: "healthy", ... } # Or: Error if backend not configured (OK for now) # Test orchestrator plugin (reads local files) orch status # Expected: { active_tasks: 0, completed_tasks: 0, health: "healthy" } # Or: Error if orchestrator not started yet (OK for now) # Test auth plugin (requires control center) auth verify # Expected: { active: false } # Or: Error if control center not running (OK for now) ``` **Note**: It's OK if plugins show errors at this stage. We'll configure backends and services later. ### Skip Plugins (Not Recommended) If you want to skip plugin installation for now: - ✅ All features work via HTTP API (slower but functional) - ⚠️ You'll miss 10-50x performance improvements - ⚠️ No offline capability for KMS/orchestrator - ℹ️ You can install plugins later anytime To use HTTP fallback: ```bash # System automatically uses HTTP if plugins not available # No configuration changes needed ``` --- ## Step 3: Install Required Tools ### Essential Tools **SOPS (Secrets Management)** ```bash # macOS brew install sops # Linux wget https://github.com/mozilla/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64 sudo mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops sudo chmod +x /usr/local/bin/sops # Verify sops --version # Expected: 3.10.2 or higher ``` **Age (Encryption Tool)** ```bash # macOS brew install age # Linux sudo apt install age # Ubuntu/Debian sudo dnf install age # Fedora # Or from source go install filippo.io/age/cmd/...@latest # Verify age --version # Expected: 1.2.1 or higher # Generate Age key (for local encryption) age-keygen -o ~/.age/key.txt cat ~/.age/key.txt # Save the public key (age1...) for later ``` ### Optional but Recommended Tools **K9s (Kubernetes Management)** ```bash # macOS brew install k9s # Linux curl -sS https://webinstall.dev/k9s | bash # Verify k9s version # Expected: 0.50.6 or higher ``` **glow (Markdown Renderer)** ```bash # macOS brew install glow # Linux sudo apt install glow # Ubuntu/Debian sudo dnf install glow # Fedora # Verify glow --version ``` --- ## Step 4: Clone and Setup Project ### Clone Repository ```bash # Clone project git clone https://github.com/your-org/project-provisioning.git cd project-provisioning # Or if already cloned, update to latest git pull origin main ``` ### Add CLI to PATH (Optional) ```bash # Add to ~/.bashrc or ~/.zshrc export PATH="$PATH:/Users/Akasha/project-provisioning/provisioning/core/cli" # Or create symlink sudo ln -s /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning /usr/local/bin/provisioning # Verify provisioning version # Expected: 3.5.0 ``` --- ## Step 5: Initialize Workspace A workspace is a self-contained environment for managing infrastructure. ### Create New Workspace ```bash # Initialize new workspace provisioning workspace init --name production # Or use interactive mode provisioning workspace init # Name: production # Description: Production infrastructure # Provider: upcloud ``` **What this creates:** The new workspace initialization now generates **Nickel configuration files** for type-safe, schema-validated infrastructure definitions: ```plaintext workspace/ ├── config/ │ ├── config.ncl # Master Nickel configuration (type-safe) │ ├── providers/ │ │ └── upcloud.toml # Provider-specific settings │ ├── platform/ # Platform service configs │ └── kms.toml # Key management settings ├── infra/ │ └── default/ │ ├── main.ncl # Infrastructure entry point │ └── servers.ncl # Server definitions ├── docs/ # Auto-generated guides └── workspace.nu # Workspace utility scripts ``` ### Workspace Configuration Format The workspace configuration uses **Nickel (type-safe, validated)**. This provides: - ✅ **Type Safety**: Schema validation catches errors at load time - ✅ **Lazy Evaluation**: Only computes what's needed - ✅ **Validation**: Record merging, required fields, constraints - ✅ **Documentation**: Self-documenting with records **Example Nickel config** (`config.ncl`): ```nickel { workspace = { name = "production", version = "1.0.0", created = "2025-12-03T14:30:00Z", }, paths = { base = "/opt/workspaces/production", infra = "/opt/workspaces/production/infra", cache = "/opt/workspaces/production/.cache", }, providers = { active = ["upcloud"], default = "upcloud", }, } ``` ### Verify Workspace ```bash # Show workspace info provisioning workspace info # List all workspaces provisioning workspace list # Show active workspace provisioning workspace active # Expected: production ``` ### View and Validate Workspace Configuration Now you can inspect and validate your Nickel workspace configuration: ```bash # View complete workspace configuration provisioning workspace config show # Show specific workspace provisioning workspace config show production # View configuration in different formats provisioning workspace config show --format=json provisioning workspace config show --format=yaml provisioning workspace config show --format=nickel # Raw Nickel file # Validate workspace configuration provisioning workspace config validate # Output: ✅ Validation complete - all configs are valid # Show configuration hierarchy (priority order) provisioning workspace config hierarchy ``` **Configuration Validation**: The Nickel schema automatically validates: - ✅ Semantic versioning format (for example, "1.0.0") - ✅ Required sections present (workspace, paths, provisioning, etc.) - ✅ Valid file paths and types - ✅ Provider configuration exists for active providers - ✅ KMS and SOPS settings properly configured --- ## Step 6: Configure Environment ### Set Provider Credentials **UpCloud Provider:** ```bash # Create provider config vim workspace/config/providers/upcloud.toml ``` ```toml [upcloud] username = "your-upcloud-username" password = "your-upcloud-password" # Will be encrypted # Default settings default_zone = "de-fra1" default_plan = "2xCPU-4 GB" ``` **AWS Provider:** ```bash # Create AWS config vim workspace/config/providers/aws.toml ``` ```toml [aws] region = "us-east-1" access_key_id = "AKIAXXXXX" secret_access_key = "xxxxx" # Will be encrypted # Default settings default_instance_type = "t3.medium" default_region = "us-east-1" ``` ### Encrypt Sensitive Data ```bash # Generate Age key if not done already age-keygen -o ~/.age/key.txt # Encrypt provider configs kms encrypt (open workspace/config/providers/upcloud.toml) --backend age \ | save workspace/config/providers/upcloud.toml.enc # Or use SOPS sops --encrypt --age $(cat ~/.age/key.txt | grep "public key:" | cut -d: -f2) \ workspace/config/providers/upcloud.toml > workspace/config/providers/upcloud.toml.enc # Remove plaintext rm workspace/config/providers/upcloud.toml ``` ### Configure Local Overrides ```bash # Edit user-specific settings vim workspace/config/local-overrides.toml ``` ```toml [user] name = "admin" email = "admin@example.com" [preferences] editor = "vim" output_format = "yaml" confirm_delete = true confirm_deploy = true [http] use_curl = true # Use curl instead of ureq [paths] ssh_key = "~/.ssh/id_ed25519" ``` --- ## Step 7: Discover and Load Modules ### Discover Available Modules ```bash # Discover task services provisioning module discover taskserv # Shows: kubernetes, containerd, etcd, cilium, helm, etc. # Discover providers provisioning module discover provider # Shows: upcloud, aws, local # Discover clusters provisioning module discover cluster # Shows: buildkit, registry, monitoring, etc. ``` ### Load Modules into Workspace ```bash # Load Kubernetes taskserv provisioning module load taskserv production kubernetes # Load multiple modules provisioning module load taskserv production kubernetes containerd cilium # Load cluster configuration provisioning module load cluster production buildkit # Verify loaded modules provisioning module list taskserv production provisioning module list cluster production ``` --- ## Step 8: Validate Configuration Before deploying, validate all configuration: ```bash # Validate workspace configuration provisioning workspace validate # Validate infrastructure configuration provisioning validate config # Validate specific infrastructure provisioning infra validate --infra production # Check environment variables provisioning env # Show all configuration and environment provisioning allenv ``` **Expected output:** ```plaintext ✓ Configuration valid ✓ Provider credentials configured ✓ Workspace initialized ✓ Modules loaded: 3 taskservs, 1 cluster ✓ SSH key configured ✓ Age encryption key available ``` **Fix any errors** before proceeding to deployment. --- ## Step 9: Deploy Servers ### Preview Server Creation (Dry Run) ```bash # Check what would be created (no actual changes) provisioning server create --infra production --check # With debug output for details provisioning server create --infra production --check --debug ``` **Review the output:** - Server names and configurations - Zones and regions - CPU, memory, disk specifications - Estimated costs - Network settings ### Create Servers ```bash # Create servers (with confirmation prompt) provisioning server create --infra production # Or auto-confirm (skip prompt) provisioning server create --infra production --yes # Wait for completion provisioning server create --infra production --wait ``` **Expected output:** ```plaintext Creating servers for infrastructure: production ● Creating server: k8s-master-01 (de-fra1, 4xCPU-8 GB) ● Creating server: k8s-worker-01 (de-fra1, 4xCPU-8 GB) ● Creating server: k8s-worker-02 (de-fra1, 4xCPU-8 GB) ✓ Created 3 servers in 120 seconds Servers: • k8s-master-01: 192.168.1.10 (Running) • k8s-worker-01: 192.168.1.11 (Running) • k8s-worker-02: 192.168.1.12 (Running) ``` ### Verify Server Creation ```bash # List all servers provisioning server list --infra production # Show detailed server info provisioning server list --infra production --out yaml # SSH to server (test connectivity) provisioning server ssh k8s-master-01 # Type 'exit' to return ``` --- ## Step 10: Install Task Services Task services are infrastructure components like Kubernetes, databases, monitoring, etc. ### Install Kubernetes (Check Mode First) ```bash # Preview Kubernetes installation provisioning taskserv create kubernetes --infra production --check # Shows: # - Dependencies required (containerd, etcd) # - Configuration to be applied # - Resources needed # - Estimated installation time ``` ### Install Kubernetes ```bash # Install Kubernetes (with dependencies) provisioning taskserv create kubernetes --infra production # Or install dependencies first provisioning taskserv create containerd --infra production provisioning taskserv create etcd --infra production provisioning taskserv create kubernetes --infra production # Monitor progress provisioning workflow monitor ``` **Expected output:** ```plaintext Installing taskserv: kubernetes ● Installing containerd on k8s-master-01 ● Installing containerd on k8s-worker-01 ● Installing containerd on k8s-worker-02 ✓ Containerd installed (30s) ● Installing etcd on k8s-master-01 ✓ etcd installed (20s) ● Installing Kubernetes control plane on k8s-master-01 ✓ Kubernetes control plane ready (45s) ● Joining worker nodes ✓ k8s-worker-01 joined (15s) ✓ k8s-worker-02 joined (15s) ✓ Kubernetes installation complete (125 seconds) Cluster Info: • Version: 1.28.0 • Nodes: 3 (1 control-plane, 2 workers) • API Server: https://192.168.1.10:6443 ``` ### Install Additional Services ```bash # Install Cilium (CNI) provisioning taskserv create cilium --infra production # Install Helm provisioning taskserv create helm --infra production # Verify all taskservs provisioning taskserv list --infra production ``` --- ## Step 11: Create Clusters Clusters are complete application stacks (for example, BuildKit, OCI Registry, Monitoring). ### Create BuildKit Cluster (Check Mode) ```bash # Preview cluster creation provisioning cluster create buildkit --infra production --check # Shows: # - Components to be deployed # - Dependencies required # - Configuration values # - Resource requirements ``` ### Create BuildKit Cluster ```bash # Create BuildKit cluster provisioning cluster create buildkit --infra production # Monitor deployment provisioning workflow monitor # Or use plugin for faster monitoring orch tasks --status running ``` **Expected output:** ```plaintext Creating cluster: buildkit ● Deploying BuildKit daemon ● Deploying BuildKit worker ● Configuring BuildKit cache ● Setting up BuildKit registry integration ✓ BuildKit cluster ready (60 seconds) Cluster Info: • BuildKit version: 0.12.0 • Workers: 2 • Cache: 50 GB • Registry: registry.production.local ``` ### Verify Cluster ```bash # List all clusters provisioning cluster list --infra production # Show cluster details provisioning cluster list --infra production --out yaml # Check cluster health kubectl get pods -n buildkit ``` --- ## Step 12: Verify Deployment ### Comprehensive Health Check ```bash # Check orchestrator status orch status # or provisioning orchestrator status # Check all servers provisioning server list --infra production # Check all taskservs provisioning taskserv list --infra production # Check all clusters provisioning cluster list --infra production # Verify Kubernetes cluster kubectl get nodes kubectl get pods --all-namespaces ``` ### Run Validation Tests ```bash # Validate infrastructure provisioning infra validate --infra production # Test connectivity provisioning server ssh k8s-master-01 "kubectl get nodes" # Test BuildKit kubectl exec -it -n buildkit buildkit-0 -- buildctl --version ``` ### Expected Results All checks should show: - ✅ Servers: Running - ✅ Taskservs: Installed and healthy - ✅ Clusters: Deployed and operational - ✅ Kubernetes: 3/3 nodes ready - ✅ BuildKit: 2/2 workers ready --- ## Step 13: Post-Deployment ### Configure kubectl Access ```bash # Get kubeconfig from master node provisioning server ssh k8s-master-01 "cat ~/.kube/config" > ~/.kube/config-production # Set KUBECONFIG export KUBECONFIG=~/.kube/config-production # Verify access kubectl get nodes kubectl get pods --all-namespaces ``` ### Set Up Monitoring (Optional) ```bash # Deploy monitoring stack provisioning cluster create monitoring --infra production # Access Grafana kubectl port-forward -n monitoring svc/grafana 3000:80 # Open: http://localhost:3000 ``` ### Configure CI/CD Integration (Optional) ```bash # Generate CI/CD credentials provisioning secrets generate aws --ttl 12h # Create CI/CD kubeconfig kubectl create serviceaccount ci-cd -n default kubectl create clusterrolebinding ci-cd --clusterrole=admin --serviceaccount=default:ci-cd ``` ### Backup Configuration ```bash # Backup workspace configuration tar -czf workspace-production-backup.tar.gz workspace/ # Encrypt backup kms encrypt (open workspace-production-backup.tar.gz | encode base64) --backend age \ | save workspace-production-backup.tar.gz.enc # Store securely (S3, Vault, etc.) ``` --- ## Troubleshooting ### Server Creation Fails **Problem**: Server creation times out or fails ```bash # Check provider credentials provisioning validate config # Check provider API status curl -u username:password https://api.upcloud.com/1.3/account # Try with debug mode provisioning server create --infra production --check --debug ``` ### Taskserv Installation Fails **Problem**: Kubernetes installation fails ```bash # Check server connectivity provisioning server ssh k8s-master-01 # Check logs provisioning orchestrator logs | grep kubernetes # Check dependencies provisioning taskserv list --infra production | where status == "failed" # Retry installation provisioning taskserv delete kubernetes --infra production provisioning taskserv create kubernetes --infra production ``` ### Plugin Commands Don't Work **Problem**: `auth`, `kms`, or `orch` commands not found ```bash # Check plugin registration plugin list | where name =~ "auth|kms|orch" # Re-register if missing cd provisioning/core/plugins/nushell-plugins plugin add target/release/nu_plugin_auth plugin add target/release/nu_plugin_kms plugin add target/release/nu_plugin_orchestrator # Restart Nushell exit nu ``` ### KMS Encryption Fails **Problem**: `kms encrypt` returns error ```bash # Check backend status kms status # Check RustyVault running curl http://localhost:8200/v1/sys/health # Use Age backend instead (local) kms encrypt "data" --backend age --key age1xxxxxxxxx # Check Age key cat ~/.age/key.txt ``` ### Orchestrator Not Running **Problem**: `orch status` returns error ```bash # Check orchestrator status ps aux | grep orchestrator # Start orchestrator cd provisioning/platform/orchestrator ./scripts/start-orchestrator.nu --background # Check logs tail -f provisioning/platform/orchestrator/data/orchestrator.log ``` ### Configuration Validation Errors **Problem**: `provisioning validate config` shows errors ```bash # Show detailed errors provisioning validate config --debug # Check configuration files provisioning allenv # Fix missing settings vim workspace/config/local-overrides.toml ``` --- ## Next Steps ### Explore Advanced Features 1. **Multi-Environment Deployment** ```bash # Create dev and staging workspaces provisioning workspace create dev provisioning workspace create staging provisioning workspace switch dev ``` 1. **Batch Operations** ```bash # Deploy to multiple clouds provisioning batch submit workflows/multi-cloud-deploy.ncl ``` 2. **Security Features** ```bash # Enable MFA auth mfa enroll totp # Set up break-glass provisioning break-glass request "Emergency access" ``` 3. **Compliance and Audit** ```bash # Generate compliance report provisioning compliance report --standard soc2 ``` ### Learn More - **Quick Reference**: `provisioning sc` or `docs/guides/quickstart-cheatsheet.md` - **Update Guide**: `docs/guides/update-infrastructure.md` - **Customize Guide**: `docs/guides/customize-infrastructure.md` - **Plugin Guide**: `docs/user/PLUGIN_INTEGRATION_GUIDE.md` - **Security System**: `docs/architecture/adr-009-security-system-complete.md` ### Get Help ```bash # Show help for any command provisioning help provisioning help server provisioning help taskserv # Check version provisioning version # Start Nushell session with provisioning library provisioning nu ``` --- ## Summary You've successfully: ✅ Installed Nushell and essential tools ✅ Built and registered native plugins (10-50x faster operations) ✅ Cloned and configured the project ✅ Initialized a production workspace ✅ Configured provider credentials ✅ Deployed servers ✅ Installed Kubernetes and task services ✅ Created application clusters ✅ Verified complete deployment **Your infrastructure is now ready for production use!** --- **Estimated Total Time**: 30-60 minutes **Next Guide**: [Update Infrastructure](update-infrastructure.md) **Questions?**: Open an issue or contact **Last Updated**: 2025-10-09 **Version**: 3.5.0