# Verification\n\nThis guide helps you verify that your Provisioning Platform deployment is working correctly.\n\n## Overview\n\nAfter completing your first deployment, verify:\n\n1. System configuration\n2. Server accessibility\n3. Task service health\n4. Platform services (if installed)\n\n## Step 1: Verify Configuration\n\nCheck that all configuration is valid:\n\n```\n# Validate all configuration\nprovisioning validate config\n\n# Expected output:\n# ✓ Configuration valid\n# ✓ No errors found\n# ✓ All required fields present\n```\n\n```\n# Check environment variables\nprovisioning env\n\n# View complete configuration\nprovisioning allenv\n```\n\n## Step 2: Verify Servers\n\nCheck that servers are accessible and healthy:\n\n```\n# List all servers\nprovisioning server list\n\n# Expected output:\n# ┌───────────────┬──────────┬───────┬────────┬──────────────┬──────────┐\n# │ Hostname      │ Provider │ Cores │ Memory │ IP Address   │ Status   │\n# ├───────────────┼──────────┼───────┼────────┼──────────────┼──────────┤\n# │ dev-server-01 │ local    │ 2     │ 4096   │ 192.168.1.100│ running  │\n# └───────────────┴──────────┴───────┴────────┴──────────────┴──────────┘\n```\n\n```\n# Check server details\nprovisioning server info dev-server-01\n\n# Test SSH connectivity\nprovisioning server ssh dev-server-01 -- echo "SSH working"\n```\n\n## Step 3: Verify Task Services\n\nCheck installed task services:\n\n```\n# List task services\nprovisioning taskserv list\n\n# Expected output:\n# ┌────────────┬─────────┬────────────────┬──────────┐\n# │ Name       │ Version │ Server         │ Status   │\n# ├────────────┼─────────┼────────────────┼──────────┤\n# │ containerd │ 1.7.0   │ dev-server-01  │ running  │\n# │ etcd       │ 3.5.0   │ dev-server-01  │ running  │\n# │ kubernetes │ 1.28.0  │ dev-server-01  │ running  │\n# └────────────┴─────────┴────────────────┴──────────┘\n```\n\n```\n# Check specific task service\nprovisioning taskserv status kubernetes\n\n# View task service logs\nprovisioning taskserv logs kubernetes --tail 50\n```\n\n## Step 4: Verify Kubernetes (If Installed)\n\nIf you installed Kubernetes, verify it's working:\n\n```\n# Check Kubernetes nodes\nprovisioning server ssh dev-server-01 -- kubectl get nodes\n\n# Expected output:\n# NAME            STATUS   ROLES           AGE   VERSION\n# dev-server-01   Ready    control-plane   10m   v1.28.0\n```\n\n```\n# Check Kubernetes pods\nprovisioning server ssh dev-server-01 -- kubectl get pods -A\n\n# All pods should be Running or Completed\n```\n\n## Step 5: Verify Platform Services (Optional)\n\nIf you installed platform services:\n\n### Orchestrator\n\n```\n# Check orchestrator health\ncurl http://localhost:8080/health\n\n# Expected:\n# {"status":"healthy","version":"0.1.0"}\n```\n\n```\n# List tasks\ncurl http://localhost:8080/tasks\n```\n\n### Control Center\n\n```\n# Check control center health\ncurl http://localhost:9090/health\n\n# Test policy evaluation\ncurl -X POST http://localhost:9090/policies/evaluate \\n  -H "Content-Type: application/json" \\n  -d '{"principal":{"id":"test"},"action":{"id":"read"},"resource":{"id":"test"}}'\n```\n\n### KMS Service\n\n```\n# Check KMS health\ncurl http://localhost:8082/api/v1/kms/health\n\n# Test encryption\necho "test" | provisioning kms encrypt\n```\n\n## Step 6: Run Health Checks\n\nRun comprehensive health checks:\n\n```\n# Check all components\nprovisioning health check\n\n# Expected output:\n# ✓ Configuration: OK\n# ✓ Servers: 1/1 healthy\n# ✓ Task Services: 3/3 running\n# ✓ Platform Services: 3/3 healthy\n# ✓ Network Connectivity: OK\n# ✓ Encryption Keys: OK\n```\n\n## Step 7: Verify Workflows\n\nIf you used workflows:\n\n```\n# List all workflows\nprovisioning workflow list\n\n# Check specific workflow\nprovisioning workflow status <workflow-id>\n\n# View workflow stats\nprovisioning workflow stats\n```\n\n## Common Verification Checks\n\n### DNS Resolution (If CoreDNS Installed)\n\n```\n# Test DNS resolution\ndig @localhost test.provisioning.local\n\n# Check CoreDNS status\nprovisioning server ssh dev-server-01 -- systemctl status coredns\n```\n\n### Network Connectivity\n\n```\n# Test server-to-server connectivity\nprovisioning server ssh dev-server-01 -- ping -c 3 dev-server-02\n\n# Check firewall rules\nprovisioning server ssh dev-server-01 -- sudo iptables -L\n```\n\n### Storage and Resources\n\n```\n# Check disk usage\nprovisioning server ssh dev-server-01 -- df -h\n\n# Check memory usage\nprovisioning server ssh dev-server-01 -- free -h\n\n# Check CPU usage\nprovisioning server ssh dev-server-01 -- top -bn1 | head -20\n```\n\n## Troubleshooting Failed Verifications\n\n### Configuration Validation Failed\n\n```\n# View detailed error\nprovisioning validate config --verbose\n\n# Check specific infrastructure\nprovisioning validate config --infra my-infra\n```\n\n### Server Unreachable\n\n```\n# Check server logs\nprovisioning server logs dev-server-01\n\n# Try debug mode\nprovisioning --debug server ssh dev-server-01\n```\n\n### Task Service Not Running\n\n```\n# Check service logs\nprovisioning taskserv logs kubernetes\n\n# Restart service\nprovisioning taskserv restart kubernetes --infra my-infra\n```\n\n### Platform Service Down\n\n```\n# Check service status\nprovisioning platform status orchestrator\n\n# View service logs\nprovisioning platform logs orchestrator --tail 100\n\n# Restart service\nprovisioning platform restart orchestrator\n```\n\n## Performance Verification\n\n### Response Time Tests\n\n```\n# Measure server response time\ntime provisioning server info dev-server-01\n\n# Measure task service response time\ntime provisioning taskserv list\n\n# Measure workflow submission time\ntime provisioning workflow submit test-workflow.ncl\n```\n\n### Resource Usage\n\n```\n# Check platform resource usage\ndocker stats  # If using Docker\n\n# Check system resources\nprovisioning system resources\n```\n\n## Security Verification\n\n### Encryption\n\n```\n# Verify encryption keys\nls -la ~/.config/provisioning/age/\n\n# Test encryption/decryption\necho "test" | provisioning kms encrypt | provisioning kms decrypt\n```\n\n### Authentication (If Enabled)\n\n```\n# Test login\nprovisioning login --username admin\n\n# Verify token\nprovisioning whoami\n\n# Test MFA (if enabled)\nprovisioning mfa verify <code>\n```\n\n## Verification Checklist\n\nUse this checklist to ensure everything is working:\n\n- [ ] Configuration validation passes\n- [ ] All servers are accessible via SSH\n- [ ] All servers show "running" status\n- [ ] All task services show "running" status\n- [ ] Kubernetes nodes are "Ready" (if installed)\n- [ ] Kubernetes pods are "Running" (if installed)\n- [ ] Platform services respond to health checks\n- [ ] Encryption/decryption works\n- [ ] Workflows can be submitted and complete\n- [ ] No errors in logs\n- [ ] Resource usage is within expected limits\n\n## Next Steps\n\nOnce verification is complete:\n\n- **[User Guide](../user/README.md)** - Learn advanced features\n- **[Quick Reference](../guides/quickstart-cheatsheet.md)** - Command shortcuts\n- **[Infrastructure Management](../user/infrastructure-management.md)** - Day-to-day operations\n- **[Troubleshooting](../user/troubleshooting-guide.md)** - Common issues and solutions\n\n## Additional Resources\n\n- [Complete From-Scratch Guide](../guides/from-scratch.md)\n- [Service Management Guide](../user/SERVICE_MANAGEMENT_GUIDE.md)\n- [Test Environment Guide](../user/test-environment-guide.md)\n\n---\n\n**Congratulations!** You've successfully deployed and verified your first Provisioning Platform infrastructure!
