provisioning/docs/src/getting-started/04-verification.md

# Verification

This guide helps you verify that your Provisioning Platform deployment is working correctly.

## Overview

After completing your first deployment, verify:

1. System configuration
2. Server accessibility
3. Task service health
4. Platform services (if installed)

## Step 1: Verify Configuration

Check that all configuration is valid:

```bash
# Validate all configuration
provisioning validate config

# Expected output:
# ✓ Configuration valid
# ✓ No errors found
# ✓ All required fields present
```

```bash
# Check environment variables
provisioning env

# View complete configuration
provisioning allenv
```

## Step 2: Verify Servers

Check that servers are accessible and healthy:

```bash
# List all servers
provisioning server list

# Expected output:
# ┌───────────────┬──────────┬───────┬────────┬──────────────┬──────────┐
# │ Hostname      │ Provider │ Cores │ Memory │ IP Address   │ Status   │
# ├───────────────┼──────────┼───────┼────────┼──────────────┼──────────┤
# │ dev-server-01 │ local    │ 2     │ 4096   │ 192.168.1.100│ running  │
# └───────────────┴──────────┴───────┴────────┴──────────────┴──────────┘
```

```bash
# Check server details
provisioning server info dev-server-01

# Test SSH connectivity
provisioning server ssh dev-server-01 -- echo "SSH working"
```

## Step 3: Verify Task Services

Check installed task services:

```bash
# List task services
provisioning taskserv list

# Expected output:
# ┌────────────┬─────────┬────────────────┬──────────┐
# │ Name       │ Version │ Server         │ Status   │
# ├────────────┼─────────┼────────────────┼──────────┤
# │ containerd │ 1.7.0   │ dev-server-01  │ running  │
# │ etcd       │ 3.5.0   │ dev-server-01  │ running  │
# │ kubernetes │ 1.28.0  │ dev-server-01  │ running  │
# └────────────┴─────────┴────────────────┴──────────┘
```

```bash
# Check specific task service
provisioning taskserv status kubernetes

# View task service logs
provisioning taskserv logs kubernetes --tail 50
```

## Step 4: Verify Kubernetes (If Installed)

If you installed Kubernetes, verify it's working:

```bash
# Check Kubernetes nodes
provisioning server ssh dev-server-01 -- kubectl get nodes

# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# dev-server-01   Ready    control-plane   10m   v1.28.0
```

```bash
# Check Kubernetes pods
provisioning server ssh dev-server-01 -- kubectl get pods -A

# All pods should be Running or Completed
```

## Step 5: Verify Platform Services (Optional)

If you installed platform services:

### Orchestrator

```bash
# Check orchestrator health
curl http://localhost:8080/health

# Expected:
# {"status":"healthy","version":"0.1.0"}
```

```bash
# List tasks
curl http://localhost:8080/tasks
```

### Control Center

```bash
# Check control center health
curl http://localhost:9090/health

# Test policy evaluation
curl -X POST http://localhost:9090/policies/evaluate \
  -H "Content-Type: application/json" \
  -d '{"principal":{"id":"test"},"action":{"id":"read"},"resource":{"id":"test"}}'
```

### KMS Service

```bash
# Check KMS health
curl http://localhost:8082/api/v1/kms/health

# Test encryption
echo "test" | provisioning kms encrypt
```

## Step 6: Run Health Checks

Run comprehensive health checks:

```bash
# Check all components
provisioning health check

# Expected output:
# ✓ Configuration: OK
# ✓ Servers: 1/1 healthy
# ✓ Task Services: 3/3 running
# ✓ Platform Services: 3/3 healthy
# ✓ Network Connectivity: OK
# ✓ Encryption Keys: OK
```

## Step 7: Verify Workflows

If you used workflows:

```bash
# List all workflows
provisioning workflow list

# Check specific workflow
provisioning workflow status <workflow-id>

# View workflow stats
provisioning workflow stats
```

## Common Verification Checks

### DNS Resolution (If CoreDNS Installed)

```bash
# Test DNS resolution
dig @localhost test.provisioning.local

# Check CoreDNS status
provisioning server ssh dev-server-01 -- systemctl status coredns
```

### Network Connectivity

```bash
# Test server-to-server connectivity
provisioning server ssh dev-server-01 -- ping -c 3 dev-server-02

# Check firewall rules
provisioning server ssh dev-server-01 -- sudo iptables -L
```

### Storage and Resources

```bash
# Check disk usage
provisioning server ssh dev-server-01 -- df -h

# Check memory usage
provisioning server ssh dev-server-01 -- free -h

# Check CPU usage
provisioning server ssh dev-server-01 -- top -bn1 | head -20
```

## Troubleshooting Failed Verifications

### Configuration Validation Failed

```bash
# View detailed error
provisioning validate config --verbose

# Check specific infrastructure
provisioning validate config --infra my-infra
```

### Server Unreachable

```bash
# Check server logs
provisioning server logs dev-server-01

# Try debug mode
provisioning --debug server ssh dev-server-01
```

### Task Service Not Running

```bash
# Check service logs
provisioning taskserv logs kubernetes

# Restart service
provisioning taskserv restart kubernetes --infra my-infra
```

### Platform Service Down

```bash
# Check service status
provisioning platform status orchestrator

# View service logs
provisioning platform logs orchestrator --tail 100

# Restart service
provisioning platform restart orchestrator
```

## Performance Verification

### Response Time Tests

```bash
# Measure server response time
time provisioning server info dev-server-01

# Measure task service response time
time provisioning taskserv list

# Measure workflow submission time
time provisioning workflow submit test-workflow.ncl
```

### Resource Usage

```bash
# Check platform resource usage
docker stats  # If using Docker

# Check system resources
provisioning system resources
```

## Security Verification

### Encryption

```bash
# Verify encryption keys
ls -la ~/.config/provisioning/age/

# Test encryption/decryption
echo "test" | provisioning kms encrypt | provisioning kms decrypt
```

### Authentication (If Enabled)

```bash
# Test login
provisioning login --username admin

# Verify token
provisioning whoami

# Test MFA (if enabled)
provisioning mfa verify <code>
```

## Verification Checklist

Use this checklist to ensure everything is working:

- [ ] Configuration validation passes
- [ ] All servers are accessible via SSH
- [ ] All servers show "running" status
- [ ] All task services show "running" status
- [ ] Kubernetes nodes are "Ready" (if installed)
- [ ] Kubernetes pods are "Running" (if installed)
- [ ] Platform services respond to health checks
- [ ] Encryption/decryption works
- [ ] Workflows can be submitted and complete
- [ ] No errors in logs
- [ ] Resource usage is within expected limits

## Next Steps

Once verification is complete:

- **[User Guide](../user/README.md)** - Learn advanced features
- **[Quick Reference](../guides/quickstart-cheatsheet.md)** - Command shortcuts
- **[Infrastructure Management](../user/infrastructure-management.md)** - Day-to-day operations
- **[Troubleshooting](../user/troubleshooting-guide.md)** - Common issues and solutions

## Additional Resources

- [Complete From-Scratch Guide](../guides/from-scratch.md)
- [Service Management Guide](../user/SERVICE_MANAGEMENT_GUIDE.md)
- [Test Environment Guide](../user/test-environment-guide.md)

---

**Congratulations!** You've successfully deployed and verified your first Provisioning Platform infrastructure!