# Platform Health

Health monitoring, status checks, and system integrity validation for the Provisioning platform.

## Health Check Overview

The platform provides multiple levels of health monitoring:

| Level | Scope | Frequency | Response Time |
| ----- | ----- | --------- | ------------- |
| **Service Health** | Individual service status | Every 10s | < 100ms |
| **System Health** | Overall platform status | Every 30s | < 500ms |
| **Infrastructure Health** | Managed resources | Every 60s | < 2s |
| **Dependency Health** | External services | Every 60s | < 1s |

## Quick Health Check

```bash
# Check overall platform health
provisioning health

# Output:
# ✓ Orchestrator: healthy (uptime: 5d 3h)
# ✓ Control Center: healthy
# ✓ Vault Service: healthy
# ✓ Database: healthy (connections: 45/100)
# ✓ Network: healthy
# ✗ MCP Server: degraded (high latency)

# Exit code: 0 = healthy, 1 = degraded, 2 = unhealthy
```

## Service Health Endpoints

All services expose `/health` endpoints returning standardized responses.

### Orchestrator Health

```bash
curl  [http://localhost:8080/health](http://localhost:8080/health)
```

```json
{
  "status": "healthy",
  "version": "5.0.0",
  "uptime_seconds": 432000,
  "checks": {
    "database": "healthy",
    "file_system": "healthy",
    "memory": "healthy"
  },
  "metrics": {
    "active_workflows": 12,
    "queued_tasks": 45,
    "completed_tasks": 9876,
    "worker_threads": 8
  },
  "timestamp": "2026-01-16T10:30:00Z"
}
```

Health status values:
- `healthy` - Service operating normally
- `degraded` - Service functional with reduced capacity
- `unhealthy` - Service not functioning

### Control Center Health

```bash
curl  [http://localhost:8081/health](http://localhost:8081/health)
```

```json
{
  "status": "healthy",
  "version": "5.0.0",
  "checks": {
    "database": "healthy",
    "orchestrator": "healthy",
    "vault": "healthy",
    "auth": "healthy"
  },
  "metrics": {
    "active_sessions": 23,
    "api_requests_per_second": 156,
    "p95_latency_ms": 45
  }
}
```

### Vault Service Health

```bash
curl  [http://localhost:8085/health](http://localhost:8085/health)
```

```json
{
  "status": "healthy",
  "checks": {
    "kms_backend": "healthy",
    "encryption": "healthy",
    "key_rotation": "healthy"
  },
  "metrics": {
    "active_secrets": 234,
    "encryption_ops_per_second": 50,
    "kms_latency_ms": 3
  }
}
```

## System Health Checks

### Comprehensive Health Check

```bash
# Run all health checks
provisioning health check --all

# Check specific components
provisioning health check --components orchestrator,database,network

# Output detailed report
provisioning health check --detailed --output /tmp/health-report.json
```

### Health Check Components

Platform health checking verifies:

1. **Service Availability** - All services responding
2. **Database Connectivity** - SurrealDB reachable and responsive
3. **Filesystem Health** - Disk space and I/O performance
4. **Network Connectivity** - Internal and external connectivity
5. **Resource Utilization** - CPU, memory, disk within limits
6. **Dependency Status** - External services available
7. **Security Status** - Authentication and encryption functional

### Database Health

```bash
# Check database health
provisioning health database

# Output:
# ✓ Connection: healthy (latency: 2ms)
# ✓ Disk usage: 45% (22GB / 50GB)
# ✓ Active connections: 45 / 100
# ✓ Query performance: healthy (avg: 15ms)
# ✗ Replication: warning (lag: 5s)
```

Detailed database metrics:

```bash
# Connection pool status
provisioning database pool-status

# Slow query analysis
provisioning database slow-queries --threshold 1000ms

# Storage usage
provisioning database storage-stats
```

### Filesystem Health

```bash
# Check disk space and I/O
provisioning health filesystem

# Output:
# ✓ Root filesystem: 65% used (325GB / 500GB)
# ✓ Data filesystem: 45% used (225GB / 500GB)
# ✓ I/O latency: healthy (avg: 5ms)
# ✗ Inodes: warning (85% used)
```

Check specific paths:

```bash
# Check data directory
df -h /var/lib/provisioning

# Check I/O performance
iostat -x 1 5
```

### Network Health

```bash
# Check network connectivity
provisioning health network

# Test external connectivity
provisioning health network --external

# Test provider connectivity
provisioning health network --provider upcloud
```

Network health checks:

- Internal service-to-service connectivity
- DNS resolution
- External API reachability (cloud providers)
- Network latency and packet loss
- Firewall rules validation

## Resource Monitoring

### CPU Health

```bash
# Check CPU utilization
provisioning health cpu

# Per-service CPU usage
provisioning platform metrics --metric cpu_usage

# Alert if CPU > 90% for 5 minutes
```

Monitor CPU load:

```bash
# System load average
uptime

# Per-process CPU
top -b -n 1 | grep provisioning
```

### Memory Health

```bash
# Check memory utilization
provisioning health memory

# Memory breakdown by service
provisioning platform metrics --metric memory_usage

# Detect memory leaks
provisioning health memory --leak-detection
```

Memory metrics:

```bash
# Available memory
free -h

# Per-service memory
ps aux | grep provisioning | awk '{sum+=$6} END {print sum/1024 " MB"}'
```

### Disk Health

```bash
# Check disk health
provisioning health disk

# SMART status (if available)
sudo smartctl -H /dev/sda
```

## Automated Health Monitoring

### Health Check Service

Enable continuous health monitoring:

```bash
# Start health monitor
provisioning health monitor --interval 30

# Monitor with alerts
provisioning health monitor --interval 30 --alert-email [ops@example.com](mailto:ops@example.com)

# Monitor specific components
provisioning health monitor --components orchestrator,database --interval 10
```

### Systemd Health Monitoring

Systemd watchdog for automatic restart on failure:

```ini
# /etc/systemd/system/provisioning-orchestrator.service
[Service]
Type=notify
WatchdogSec=30
Restart=on-failure
RestartSec=10
StartLimitIntervalSec=300
StartLimitBurst=5
```

Service sends periodic health status:

```rust
// Rust service code
sd_notify::notify(true, &[NotifyState::Watchdog])?;
```

## Health Dashboards

### Grafana Health Dashboard

Import platform health dashboard:

```bash
provisioning monitoring install-dashboard --name platform-health
```

Dashboard panels:

- Service status indicators
- Resource utilization gauges
- Error rate graphs
- Latency histograms
- Workflow success rate
- Database connection pool

Access: [http://localhost:3000/d/platform-health](http://localhost:3000/d/platform-health)

### CLI Health Dashboard

Real-time health monitoring in terminal:

```bash
# Interactive health dashboard
provisioning health dashboard

# Auto-refresh every 5 seconds
provisioning health dashboard --refresh 5
```

## Health Alerts

### Prometheus Alert Rules

```yaml
# Platform health alerts
groups:
  - name: platform_health
    rules:
      - alert: ServiceUnhealthy
        expr: up{job=~"provisioning-.*"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service is unhealthy"

      - alert: HighMemoryUsage
        expr: process_resident_memory_bytes > 4e9
        for: 5m
        labels:
          severity: warning

      - alert: DatabaseConnectionPoolExhausted
        expr: database_connection_pool_active / database_connection_pool_max > 0.9
        for: 2m
        labels:
          severity: critical
```

### Health Check Notifications

Configure health check notifications:

```toml
# /etc/provisioning/health.toml
[notifications]
enabled = true

[notifications.email]
enabled = true
smtp_server = "smtp.example.com"
from = "[health@provisioning.example.com](mailto:health@provisioning.example.com)"
to = ["[ops@example.com](mailto:ops@example.com)"]

[notifications.slack]
enabled = true
webhook_url = " [https://hooks.slack.com/services/..."](https://hooks.slack.com/services/...")
channel = "#provisioning-health"

[notifications.pagerduty]
enabled = true
service_key = "..."
```

## Dependency Health

### External Service Health

Check health of dependencies:

```bash
# Check cloud provider API
provisioning health dependency upcloud

# Check vault service
provisioning health dependency vault

# Check all dependencies
provisioning health dependency --all
```

Dependency health includes:

- API reachability
- Authentication validity
- API quota/rate limits
- Service degradation status

### Third-party Service Monitoring

Monitor integrated services:

```bash
# Kubernetes cluster health (if managing K8s)
provisioning health kubernetes

# Database replication health
provisioning health database --replication

# Secret store health
provisioning health secrets
```

## Health Metrics

Key metrics tracked for health monitoring:

### Service Metrics

```text
provisioning_service_up{service="orchestrator"} 1
provisioning_service_health_status{service="orchestrator"} 1
provisioning_service_uptime_seconds{service="orchestrator"} 432000
```

### Resource Metrics

```text
provisioning_cpu_usage_percent 45
provisioning_memory_usage_bytes 2.5e9
provisioning_disk_usage_percent{mount="/var/lib/provisioning"} 45
provisioning_network_errors_total 0
```

### Performance Metrics

```text
provisioning_api_latency_p50_ms 25
provisioning_api_latency_p95_ms 85
provisioning_api_latency_p99_ms 150
provisioning_workflow_duration_seconds 45
```

## Health Best Practices

- Monitor all critical services continuously
- Set appropriate alert thresholds
- Test alert notifications regularly
- Maintain health check runbooks
- Review health metrics weekly
- Establish health baselines
- Automate remediation where possible
- Document health status definitions
- Integrate health checks with CI/CD
- Monitor upstream dependencies

## Troubleshooting Unhealthy State

When health check fails:

```bash
# 1. Identify unhealthy component
provisioning health check --detailed

# 2. View component logs
journalctl -u provisioning-<component> -n 100

# 3. Check resource availability
provisioning health resources

# 4. Restart unhealthy service
sudo systemctl restart provisioning-<component>

# 5. Verify recovery
provisioning health check

# 6. Review recent changes
git log --since="1 day ago" -- /etc/provisioning/
```

## Related Documentation

- [Service Management](service-management.md) - Service lifecycle
- [Monitoring](monitoring.md) - Comprehensive monitoring
- [Troubleshooting](troubleshooting.md) - Issue resolution
- [Deployment Modes](deployment-modes.md) - Installation modes
